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Introduction 

In  recent  years,  the  U.S.  Army’s  Telemedicine  and  Advanced  Technology  Research 
Center  (TATRC)  has  been  a  strategic  partner,  providing  funding  for  research  initiatives  and 
activities  in  telesurgery.  These  successive  research  initiatives  have  evolved  from 
telementoring  to  telesurgery  to  semi-autonomous  function  of  key  medical  tasks. 

The  ability  to  place  surgical  robotics  that  can  be  manipulated  remotely  and  perform  semi- 
autonomous  functions  can  add  tremendous  value  to  military  medicine  by  providing  surgical 
care  more  immediately.  In  addition,  the  ability  to  remotely  perform  medical  tasks  like  needle 
insertion  through  semi-autonomous  supervisory  control  can  lead  to  new  approaches  in 
delivery  of  care  for  the  warfighters. 

The  purpose  of  this  TATRC-funded  research  was  to  conduct  collaborative  accelerated 
medical  technology  development  through  the  evaluation  of  two  different  robotic 
(telemanipulation)  surgical  systems  -  SRI’s  M7  and  University  of  Washington’s  (UW) 
RAVEN  in  an  extreme  environment,  using  a  telecommunications  link.  The  focus  of  this 
research  was  to  explore  techniques  and  develop  technologies  necessary  to  make  telesurgery 
and  telerobotic  supervisory  autonomous  function  more  “effective”  and  relevant  in  everyday 
surgical  practice  as  well  as  in  provision  of  life  and  limb  saving  emergency  surgery  to  an 
injured  soldier  in  the  battlefield.  This  program  also  sought  to  answer  a  fundamental  question 
related  to  the  ability  to  remotely  control  a  robot  outfitted  with  an  ultrasound  to  permit  a 
remote  surgeon  to  semi-autonomously  insert  a  needle  into  a  simulated  blood  vessel.  This  was 
successfully  accomplished  with  the  M7. 

This  TATRC-funded  research  supported  (1)  the  development  and  reconfiguration  of  both 
robotic  systems  so  that  they  could  be  deployed  in  an  extreme  environment;  (2)  the 
development  of  a  semi-autonomous  (supervisory)  capability  of  a  key  medical  task;  (3) 
evaluation  of  the  semi-autonomous  task  in  an  extreme  environment;  and  (4)  evaluation  of 
basic  surgical  task  using  a  standardized  surgical  training  system. 

This  research  was  very  successful.  Both  robots  were  deployed  and  evaluated.  The  semi- 
autonomous  function  of  the  M7  was  highlighted  during  the  American  Telemedicine 
Association  (ATA)  annual  meeting  in  Nashville,  TN.  The  results  of  the  TATRC-funded 
telesurgery  research  highlight  the  ongoing  collaboration  between  various  organizations, 
continuing  the  development  of  telesurgical  autonomous  robotics  as  a  significant  tool  for 
healthcare  delivery  in  extreme  environments,  especially  for  future  application  in  medical  care 
of  the  warfighter.  Data  analysis  of  the  research  data  is  underway. 

This  document  serves  as  the  final  report  of  this  telesurgery  research. 

Telesurgery  Background 


The  state  of  telesurgery,  those  activities  that  have  been  performed  during  the  past  several 
years,  is  the  direct  result  of  TATRC’s  involvement.  Each  TATRC-funded  activity  has  led  a 
progressive  chronology  of  telementoring  to  telesurgery  to  semi-autonomous  function.  The 
aim  of  this  research  has  been  to  push  the  development  of  surgical  robotics  to  provide  expert 
care  to  injured  warfighters  on  the  battlefield;  incorporating  telecommunications  capabilities 
and  semi-autonomous  functions.  Through  effective  and  collaborative  partnerships  with  the 
UC,  SRI,  and  UW,  and  a  series  of  research  initiatives  -  NEEMO  7,  NEEMO  9,  High  Altitude 
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and  Platforms  for  Mobile  Robotic  Telesurgery  (HAPsMRT)  -  this  evolution  has  transpired. 
This  research  has  indicated  that  there  are  communications  constraints  and  that  some  level  of 
autonomy  is  required. 

Telesurgery  research  focuses  on  evaluation  and  accelerated  development  of  robotic  and 
telecommunication  systems  required  for  robust  telesurgery  application.  TATRC  funding  has 
been  previously  targeted  toward  most  telesurgery  activities  that  have  occurred  and  which  the 
assembled  team  at  UC  has  been  involved  in.  These  efforts  include  two  previous  NEEMO 
missions  (NEEMO  7  and  NEEMO  9),  the  da  Vinci  telesurgery  activities  (led  by  Intuitive 
Surgical),  and  the  UC-led  HAPsMRT.  Several  of  these  projects  required  the  use  of  robotics 
systems  or  telemanipulation  systems  used  in  surgical  care.  The  SRI  M7  and  the  UW  RAVEN 
are  the  two,  which  have  seen  significant  investment  and  evaluation  in  telesurgery 
experiments. 

SRI,  International  M7 

In  1996,  SRI,  working  with  DARPA,  advanced  the  state-of-the-art  of  telepresence  and 
telemanipulation  when  it  designed  and  prototyped  the  M7  series  of  telerobotic  manipulators. 
This  haptics-enabled  master-slave  system  featured  an  anthropomorphic  scale,  a  large  40  cm 
on-a-side  workspace,  and  high-resolution  stereoscopic  video  cameras  and  displays.  Featuring 
seven  haptic-capable  degrees  of  freedom  (including  grip),  and  interchangeable  surgical 
instruments,  it  was  the  first  telepresence  and  telemanipulation  system  specifically  designed 
for  performing  open  trauma  surgical  procedures.  SRI  was  also  the  developer  of  the  da  Vinci 
robotic  platfonn  now  commercialized  through  Intuitive  Surgical. 

Since  the  summer  of  2005,  SRI  has  worked  to  extend  its  telerobotic  platfonn  for  field 
testing  in  extreme  environments.  SRI  (1)  invested  internal  research  and  development  funds  to 
upgrade  the  M7  platfonn  to  have  separate  master  and  slave  electronics,  (2)  wrote  software  to 
pennit  teleoperation  over  Internet  Protocol  (IP),  (3)  reduced  the  size  and  weight  of  the 
system,  and  (4)  ruggedized  the  control  electronics.  Spinning  disks  within  the  system  been 
replaced  with  solid  state  flash  memory  to  permit  operation  in  harsh  environments. 

In  December  2005,  SRI  conducted  a  series  of  long-distance  teleoperation  tests  between 
SRI’s  Menlo  Park,  CA  and  Ontario,  Canada,  to  tune  the  perfonnance  of  the  robot,  and 
identify  areas  for  improvement  of  software  and  hardware  required  to  meet  identified  mission 
objectives.  In  April  2006,  it  was  successfully  deployed  in  the  NURC  Aquarius  habitat  as  part 
of  the  TATRC-funded  NEEMO  9.  This  was  the  first  time  a  surgical  robot  had  been  used  to 
perform  simulated  surgery  in  such  a  small  and  harsh  environment.  Dr.  Mehran  Anvari, 
located  in  Hamilton,  Ontario,  Canada,  successfully  sutured  simulated  skin  located  in  the 
habitat  despite  an  introduced  latency  of  over  two  seconds.  Dr.  Broderick,  NEEMO  9 
crewmember,  served  as  a  key  researcher  in  the  Aquarius  habitat  during  this  mission.  A  two 
second  latency  impacts  a  single  suture  placement,  resulting  in  a  single  knot  tying  taking  ten 
minutes  to  accomplish. 

This  system,  used  in  NEEMO  9  could  not  support  clinical  use  due  to  the  communication 
limitations  and  latency.  Therefore,  it  became  apparent  that  semi-autonomous  function  could 
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help  ameliorate  these  issues.  This  required  further  development  of  semi-autonomous 
capabilities.  Although  short  term  solutions  will  emerge,  research  must  be  continued. 

Issues  of  communications  and  latency  require  novel  approaches  and  solutions.  UC, 
partnering  with  UW  and  others,  explored  the  use  of  unique  communication  assets,  currently 
being  used  by  the  military  in  active  theaters  of  operation.  This  TATRC-funded  research 
involved  the  use  of  an  unmanned  airborne  vehicle  (UAV). 

University  of  Washington  (UW)  RA  VEN Robot 

A  U.S.  Army-funded  research  and  development  project  at  the  UW  BioRobotics 
Laboratory  (BRL)  recently  produced  a  new  telesurgical  robot  -  RAVEN 
(http://brl.ee.washington.edu).  The  UC  and  UW  team,  lead  by  Dr.  Broderick,  was  the  first  to 
use  this  system  during  a  collaborative  research  project  entitled  HAPsMRT.  It  was 
successfully  deployed  in  the  high  desert  of  southern  California  and  remotely  operated  using 
an  UAV-based  communication  system.  This  series  of  tests  evaluated  a  mobile  robotic 
system,  deployed  in  an  extreme  environment  and  controlled  wirelessly  using  a  small  UAV 
relay  station.  This  capability  presents  tremendous  opportunity  to  support  telesurgical  care 
using  mobile  systems,  where  communication  assets  are  challenging. 

Body 


Research  Plan 


This  telesurgery  research  was  conducted  in  the  extreme  environment  of  the  National 
Undersea  Research  Center  (NURC)  Aquarius  habitat.  The  habitat  is  owned  by  the  National 
Oceanographic  and  Atmospheric  Administration  (NOAA).  It  is  located  off  the  coast  of  Key 
Largo,  FL,  and  submerged  to  a  depth  of  approximately  62  feet.  Is  serves  as  a  research  facility 
for  a  variety  of  organizations,  including  the  Department  of  Defense  (DoD).  The  U.S.  Navy 
has  used  it  extensively.  One  program  that  utilizes  this  extreme  environment  is  known  as 
NASA  Extreme  Environment  Mission  Operations  (NEEMO).  To  date,  there  have  been  13 
NEEMO  missions,  focused  on  a  wide  range  of  research.  These  missions  are  technology 
accelerators.  Technology  is  pushed  forward  faster  when  challenged  with  constraints  of 
remoteness,  limited  resources,  and  limited  technical  expertise. 

This  telesurgery  research,  funded  by  TATRC  and  reported  here,  was  conducted  during 
the  NEEMO  12  mission.  NEEMO  12  was  a  collaborative  research  and  training  mission 
involving  an  number  of  organizations,  including  TATRC,  U.S.  Navy,  U.S.  Air  Force, 
NOAA,  the  University  of  North  Carolina  at  Wilmington  (UNCW),  SRI  International,  UW 
BRL,  HaiVision,  and  the  University  of  Cincinnati  (UC)  Advanced  Center  for  Telemedicine 
and  Surgical  Innovation  (ACTSI).  The  grant  that  UC  received  from  TATRC  was  to  support 
the  use  of  the  habitat  and  to  support  the  telesurgery  research  tasks. 

The  telesurgery  research,  planned  and  conducted  aboard  the  NEEMO  12  mission,  was 
developed  and  coordinated  by  UC,  SRI  and  UW.  The  Institutional  Review  Board  (IRB)  at 
UC  and  the  Human  Protection  Research  Office  (HPRO)  Office  of  Research  Protections 
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(ORP)  at  the  U.S.  Army  Medical  Research  and  Materiel  Command  (USAMRMC)  reviewed 
and  approved  this  research.  Mr.  Charles  R.  Doam  served  as  the  PI  for  the  research  and  Dr. 
Timothy  Broderick  served  as  an  aquanaut. 

Existing  partnerships  with  SRI  International,  UW  BRL  and  HaiVision  were  leveraged  as 
well  through  other  UC-funded  initiatives. 

Science  Objectives 

1)  The  development  and  modification  of  a  robotic  surgical  system  (SRI  M7)  to 
incorporate  semi-autonomous  function  of  a  key  medical  task  (ultrasound-guided 
needle  insertion). 

2)  Evaluation  of  a  semi-autonomous  function  of  a  key  task  (ultrasound-guided  needle 
insertion)  using  the  modified  SRI  M7  platform 

3)  Evaluation  of  upgraded  software  capability  of  the  RAVEN  to  support  standardized 
surgical  training  tasks. 

4)  Evaluate  the  efficiency  of  different  Coder  /  Decoders  (CODECs)  in  telesurgery 
applications. 

5)  Further  develop  and  evaluate  the  application  of  wireless  telecommunication  and 
available  deployable  surgical  robots  in  an  extreme  environment. 

6)  To  build  upon  prior  collaboration  and  success  to  catalyze  further  development  of  key 
TATRC  relationships  and  technologies. 

Methods  -  Mission  Objectives 

A  crew  of  6  individuals  (Table  1)  consisting  of  two  non-physician  astronauts,  a  flight 
surgeon,  two  habitat  technicians,  and  a  TATRC  surgeon  aquanaut  performed  research  within 
the  Aquarius  habitat.  The  crew  remained  submerged  and  lived  in  the  saturated  environment 
for  a  period  of  1 1  days  (May  7-18,  2007).  U.S.  Navy  divers  supported  the  mission  on  a  daily 
basis. 


Table  1.  NEEMO  12  Crew 


Name 

Organization 

Heidemarie  M.  Stefanyshyn-Piper  CAPT,  USN 

NASA 

Jose  M.  Hernandez 

NASA 

Timothy  J.  Broderick,  MD 

UC  /  TATRC  IPA 

Josef  F.  Schmid,  MC,  USAF 

NASA 

James  Talacek 

NURC 

Dominic  Landucci 

NURC 

During  their  1 1  day  stay  in  the  submerged  habitat,  the  crew  conducted  a  variety  of 
research  tasks  in  telesurgery.  UC  partnered  with  SRI  and  UW  to  develop,  design  and  direct 
the  advanced  medical  technology  research  -  telesurgery  experiments.  This  included 
conducting  research  and  evaluation  on  two  different  surgical  robots  through  a 
telecommunications  link,  evaluation  of  a  semi  autonomous  task,  and  evaluating  surgical  tasks 
using  the  standardized  surgical  training  system  -  Society  of  American  Gastrointestinal 
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Endoscopic  Surgeons  (SAGES)  -  Fundamentals  of  Laparoscopic  Surgery  (FLS)  training 
protocol. 

The  TATRC  grant  that  UC  received  only  funded  the  telesurgery  activities  and  use  of  the 
habitat. 

Telesurgery  Research 

The  TATRC-funded  telesurgery  research  involved  two  different  unique  robotic  systems, 
the  SRI  M7  and  the  UW  RAVEN.  Both  systems  required  modifications,  including  hardware 
and  software  modifications  to  enable  deployment  and  operation  in  the  extreme  environment 
of  the  habitat,  and  for  the  M7,  the  ability  to  support  semi-autonomous  functions.  Both 
systems  were  evaluated  in  the  habitat  on  separate  days  during  the  mission.  RAVEN  was 
evaluated  on  mission  day  (MD)  2  and  MD  3  and  the  M7  was  evaluated  on  MD  7  and  MD  8. 
Both  were  also  connected  to  the  TATRC  booth  during  the  ATA  meeting  in  Nashville,  TN. 
The  main  thrust  was  to  evaluate  semi-autonomous  function  using  a  key  task  -  ultrasound- 
guided  needle  insertion  in  a  simulated  blood  vessel. 

Prior  to  the  actual  deployment  of  these  two  robotic  systems  to  the  NURC  in  early  May 
and  subsequent  integration  in  the  habitat,  the  UW  BRL  and  SRI  modified  and  streamlined 
their  systems  and  operations.  HaiVision  provide  use  of  the  Hail 000  Motion  Picture  Experts 
Group  (MPEG)  4  CODEC. 

M7 

Preparation 

The  M7,  which  had  been  evaluated  in  the  habitat  during  NEEMO  9,  was  updated  to 
accomplish  a  new  set  of  tasks.  The  main  task  was  to  demonstrate  the  ability  of  the  system 
(M7)  to  mitigate  the  effects  of  network  latency  through  the  conduct  of  a  semi-autonomous 
subtask,  which  was  the  ability  to  integrate  interoperative  imaging  with  needle-based  robotic 
surgical  procedures  -  ultrasound-guided 
needle  insertion  when  surgeon  and 
phantom  blood  vessel  are  separated  by  a 
great  distance.  This  was  the  main 
objective  of  this  research  task. 

The  specific  tasks  that  SRI 
performed  in  preparation  included:  (1) 

Software  and  hardware  improvements: 

SRI  modified  the  software  to  enable 
scaling  of  the  linear  motion  between 
master  and  slave  and  to  enable  the 
surgeon  to  clutch  (temporarily  disengage 
master/slave  control)  to  allow 
repositioning  to  a  more  comfortable  user 
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pose.  (2)  Restructured  control  and  communication  software.  SRI  redesigned  the  packet 
structure  used  for  robotic  control  to  reduce  the  network  bandwidth  usage  and  made  other 
code  improvements  to  perfonn  more  reliably  in  unpredictable  network  situations.  The  system 
was  designed  to  hold  its  last  position  in  the  event  of  communications  interruption.  (3)  User 
interface  improvements:  The  master  console  graphical  interface  was  modified  to  permit 
display  of  ultrasound  images  in  the  surgeon  console  and  to  support  the  designation  of  the 
target  by  the  operator  on  the  ultrasound  image.  Graphic  overlays  were  created  to  assist  the 
surgeon  in  moving  through  the  steps  of  the  autonomous  needle  insertion.  (4)  Support  for 
semi-autonomous  needle  insertion:  Software  was  created  to  perform  mapping  between  the 
2D  target  position  on  the  ultrasound  image  and  the  location  of  that  target  point  in  3D  space. 

The  M7  slave  was  outfitted  with  a  SonoSite  Ultrasound  (Sonosite,  Bothell,  WA)  probe. 
This  probe  was  attached  to  one  of  the  M7  arms.  The  other  arm  was  equipped  with  needle. 
The  objective  was  to  insert  the  needle  from  a  remote  site  using  the  ultrasound  probe  to  guide 
placement  into  a  phantom  blood  vessel.  (See  inset  photo) 

Deployment  and  Integration 

The  M7,  which  consists  of  the  robotic  arm  (slave) 
and  control  unit  (master)  was  packaged  and  shipped  - 
the  slave  to  the  NURC  in  Key  Largo,  FL  and  the 
master  to  the  Gaylord  Convention  Center  in 
Nashville,  TN. 

Once  the  slave  and  ancillary  equipment 
(ultrasound  probe,  cabling,  etc)  was  received  at  the 
NURC,  it  was  processed  (repackaged)  for  potting  to 
the  habitat.  Once  inside  the  habitat,  the  system  was 
installed  in  the  bunk  room  (see  inset  photo).  The  crew 
bunks  (6  total)  were  folded  (up  or  down)  to 
accommodate  the  placement  of  the  robot.  As 
illustrated  in  the  inset  image,  the  robot  arms  were  suspended  from  the  top  two  hunks.  A  table 
was  placed  directly  underneath  to  accommodate  the  phantom  blood  vessel.  The  schematic 
illustrates  the  layout  of  the  habitat. 


entry 
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The  master  control  unit  was  deployed  in  the  TATRC  exhibit  booth  at  the  ATA  12th 
Annual  Scientific  Meeting  and  Exhibition  at  the  Gaylord  Convention  Center  in  Nashville, 
TN.  SRI’s  Mr.  Thomas  Low  and  Mr.  Kevin  Hufford  worked  to  connect  the  master  and  slave 
through  a  ~1 ,500  mile  connection  to  the  habitat.  The  local  network  topology  (Nashville)  was 
implemented  by  a  temporary  commercial  6  megabits  per  second  (Mbps)  IP  circuit  that 
supplied  dedicated  bandwidth  to  the  TATRC  booth.  During  the  conducted  of  the  research 
tasks,  all  bandwidth  was  directed  to  the  robotic  master  to  ensure  the  most  robust  connection. 

Evaluation  in  the  Habitat 

The  system  was  evaluated  within  the  habitat  and  between  the  habitat  and  the  controllers 
installed  at  the  ATA  exhibit  in  Nashville,  TN.  The  NEEMO  12  crewmember  (Broderick) 
conducted  a  series  of  tests  on  the  robotic  system  and  the  communications  network. 

ATA  Event 


One  of  the  primary  objectives  was  to  conduct  a 
live  evaluation  of  the  semi-autonomous,  teleoperated 
needle  insertion  into  a  phantom  blood  vessel  with 
remote  image-guidance  using  an  ultrasound  probe. 
Prior  to  the  activity,  the  NEEMO  12  crew  interacted 
with  US  Army  Medical  Research  and  Materiel 
Command  (USAMRMC)  commander  -  COL 
Jonathan  Jaffin  and  TATRC  leadership  -  COL  Karl 
Friedl  and  COL  Ronald  Poropatich.  Two  surgeons 
were  supporting  the  UC/SRI  team  (Doarn,  Harnett, 
Low,  Hufford,  and  Boulet)  in  the  exhibit/demonstration  area  -  Dr.  Mehran  Anvari  and  Dr. 
Jon  Bowersox.  Dr.  Anvari,  Dr.  Bowersox,  and  Mr.  Doarn  discussed  the  overall  objective  of 
the  activity  with  the  assembled  leadership  on  the  importance  of  telesurgery.  The  team  in 
Nashville  then  successfully  conducted  the  semi-autonomous  task.  This  marked  the  first  time 
ever  a  needle  was  inserted  into  a 
phantom  blood  vessel  using  an 
ultrasound  image  guidance  by  someone 
remotely  located  from  the  needle, 
ultrasound  and  tissue. 

Semi-autonomous  Function 

The  key  research  objective  of  the  M7 
was  to  have  a  remote  surgeon  (using 
master  console)  control  the  slave  robotic 
arms,  which  had  been  modified  and 
outfitted  with  a  SonoSite  ultrasound 
probe  on  one  arm  and  a  needle  on  the 
other. 
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SRI  modified  the  M7  such  that  the  pedals  were  used  to  advance  and  retract  the  needle, 
once  the  surgeon  found  the  insertion  point  with  the  ultrasound  probe. 

SRI’s  work  was  accomplished  through  a  subaward  contract  with  UC.  A  more 
comprehensive  report  on  SRI’s  initiatives  appears  in  Appendix  C.  This  appended  report  is 
authored  by  Mr.  Low  and  Mr.  Hufford.  This  report  is  a  contractual  deliverable  from  SRI  to 
UC. 

Lessons  Learned 


1)  The  robotic  system  was  not  as  easy  to  fix  in  the  austere  environment  as  was  hoped. 

2)  The  work  flow  worked  well  with  software  and  hardware  upgrades. 

3)  The  M7  was  able  to  operate  using  (1)  a  smaller  master  console,  (2)  overcoming  joint- 
joint  control  and  therefore  enabling  clutching  by  the  operator,  (3)  scaling  to  conduct 
higher  fidelity  small  scale  motion;  and  (4)  semi-autonomous  function. 

4)  Working  toward  a  common  task  -  needle  insertion  -  industry,  academia  and 
government  overcame  a  number  of  deficiencies  in  design  and  were  able  to  conduct  a 
specific  medical  task  -  needle  insertion  in  a  blood  vessel  using  ultrasound-guided 
imaging. 

5)  The  network  was  robust  and  worked  without  significant  issues. 

6)  The  HaiVision  CODEC  worked  fine  but  had  some  poor  image  and  audio  quality 
issues. 

Recommendations 


1)  Continue  to  refine  telesurgical  and  autonomous  technologies  to  accomplish  additional 
military  relevant  medical  tasks. 

2)  Refine  tasks  associated  with  initialization  and  calibration  of  the  encoders. 

3)  Modify  software  to  minimize  dramatic  changes  in  cable  tension. 

4)  Work  all  communications  issues  out  upfront,  including  camera  placement  and 
CODEC  use. 


RAVEN 


Preparation 


The  UW  RAVEN  robot  was  originally  developed 
operated  in  a  laboratory  setting  using  U.  S.  Army 
The  TATRC-funded  and  UC-led  HAPsMRT  proj 
provided  an  opportunity  for  the  system  to  be  deployed  i 
an  extreme  environment.  This  deployment  was  in  the  hig 
desert  north  of  Los  Angeles  and  it  provided  an  opportunit 
to  demonstrate  the  robustness  of  the  system.  The  lesson 
learned  from  that  research  helped  in  the  modification  an 
preparation  of  RAVEN  for  deployment  and  operation  in  th . 
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habitat.  Through  a  contract  with  the  UC,  UW  modified  the  hardware  and  software  of  the 
RAVEN  to  permit  deployment  and  operation  in  the  habitat. 

Deployment  and  Integration 

The  RAVEN  was  shipped  to  Florida  and  prepared  for  potting.  Foam  casings  that  would 
cover  the  robot  arms  and  fit  snugly  into  a  potting  dry  bag  were  used.  During  potting  of  the 
RAVEN  to  the  habitat  on  the  weekend  prior  to  the  mission,  the  foam  collapsed  about  the 
robot  ann  and  pulled  off  one  of  the  cables.  This  was  fixed  in  the  habitat  prior  to  the  start  of 
the  mission.  Once  the  system  was  potted  down  to  the  habitat,  it  was  assembled  in  the  hunk 
room,  as  highlighted  above  with  the  M7.  The  arms  were  affixed  to  a  table  rather  than  hanging 
from  the  hunks  as  was  the  case  with  the  M7. 

A  team  of  engineers  from  UW  were  also  deployed  to  Florida  to  support  the  mission.  This 
team  consisted  of  Ms.  Diane  Friedman  and  Mr.  Mitch  Lum.  They  provided  technical  support 
during  the  entire  mission. 

UC  sent  its  RAVEN  controllers  -  Sensable  Phantom  Omnis  -  to  Nashville  to  support 
operations  and  evaluation  between  the  ATA  convention  site  and  the  NURC.  UC  also 
installed  controllers  at  the  CMC  to  support  the  outreach  event. 

Evaluation  in  the  Habitat 

Once  the  RAVEN  was  assembled  in  the  habitat,  tests  were  conducted  to  determine 
communications  connectivity  with  the  NURC  and  with  ATA  convention  site.  The  robot  was 
manipulated  from  three  different  sites  -  (1)  UW  BRU;  (2)  NURC;  and  CMC.  Data  was 
collected  on  the  motion  of  the  system,  including  position  velocity  and  torque  of  the 
manipulator  arms  (end  effectors).  Data  was  recorded  while  surgeons  perfonned  standard 
SAGES  FLS  tasks.  Data  analysis  is  underway. 

The  system  was  also  used  to  simulate  rock  manipulation,  and  it  was  used  to  support  the 
CMC  event,  where  school  children  operated  the  robot. 

CMC  Event 

Prior  to  the  mission,  teams  of  middle  school  students  from  multiple  schools  from  Ohio 
and  Indiana  competed  in  a  robotic  surgery  “competition”  at  UC’s  Center  for  Surgical 
Innovation  (CSI).  These  children  had  a  great  time  with  “hands  on”  learning  that  was  focused 
on  science,  medicine,  engineering,  and  robotics. 

During  the  mission,  these  students  met  the  crew  via  videoconference  between  the  CMC 
and  the  Aquarius  habitat.  Students  also  remotely  operated  the  RAVEN  surgical  robot  in  the 
habitat. 

Connection  to  the  CMC  used  the  same  configuration  as  the  M7  did  to  Nashville.  Since 
the  CMC  had  only  a  T-l  (1.54  Mbps)  IP  network  link,  the  local  carrier,  Cincinnati  Bell, 
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reconfigured  the  connection  to  a  “burstable”  LAN  Advantage  connection  with  a  maximum 
bandwidth  of  5  Mbps.  This  permitted  a  more  robust  network. 

AT  A  Event 

The  RAVEN  controllers  -  Phantom  Omnis 
-  were  deployed  to  the  Gaylord  Convention 
Center  to  support  demonstration  of  the  ability 
to  manipulate  the  RAVEN.  Small  foam  blocks 
were  set  up  on  a  table  at  the  NURC.  The  users 
at  the  controls  in  Nashville  were  able  to 
manipulate  the  RAVEN  arms,  located  at  the 
NURC.  The  foam  samples  were  picked  up  and 
moved  around. 

UW’s  work  was  accomplished  through  a 
subaward  contract  with  UC.  A  more 
comprehensive  report  on  UW’s  initiatives 
appears  in  Appendix  D.  This  appended  report  is  authored  by  Dr.  Blake  Hannaford  and  his 
team  at  UW.  This  report  covers  the  contracted  work  that  UW  performed. 

Lessons  Learned 


1)  The  RAVEN  was  difficult  and  bulky  to  deploy.  This  required  a  UW  engineer  to  dive 
into  the  habitat  to  make  repairs. 

2)  It  took  some  time  to  set  up 

3)  The  network  was  robust  and  worked  without  significant  issues. 

4)  The  HaiVision  CODEC  worked  fine  but  had  some  poor  image  and  audio  quality 
issues. 

Recommendations 


1)  Continue  to  refine  robotic  and  telesurgery  technology  to  accomplish  additional 
military  relevant  medical  tasks. 

2)  Continue  to  refine  steps  to  decrease  deployment  time. 

3)  Work  all  communications  issues  out  upfront,  including  camera  placement  and 
CODEC  use. 

Telesurgery  Telecommunications  Network 

The  NEEMO  12  mission  used  the  same  local  NURC  network  used  in  prior  NEEMO 
missions.  The  network  was  improved  with  upgraded  Cisco  equipment.  Once  communications 
reached  the  local  Internet  Service  Provider  (ISP)  external  to  the  NURC,  it  was  completely 
different  than  prior  NEEMO  missions. 
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The  Aquarius  is  a  subnet  on  a  Local  Area  Network  (LAN)  of  the  NURC.  The  connection 
from  the  mainland  to  the  Aquarius  is  facilitated  by  a  Spectra  5.4  GHz  Wireless  Bridge/ 
minimum  30  Mbps  link  from  an  antenna  on  the  roof  of  the  facility  to  the  Life  Support  Buoy 
(LSB)  (floating  in  the  ocean  just  above  the  habitat)  where  a  Cisco  switch  is  located.  An 
umbilical  connects  the  network  and  other  life  support  cables  to  the  Aquarius.  The  NURC  is 
connected  to  the  Internet  by  TerraNova,  a  commercial  ISP  where  standard  Internet  Protocol 
(IP)  routing  takes  place.  The  network  communications  for  the  command  and  control  of  both 
robots  was  facilitated  using  Transmission  Control  Protocol/IP  (TCP/IP)  by  way  of  this 
topology.  The  audio  and  video  for  the  robotic  sessions  was  also  transported  using  this 
topology  and  related  protocols  such  as  UDP  (User  Datagram  Protocol)  and  MPEG4.  Because 
there  were  two  robotic  platforms  for  NEEMO  12,  it  was  imperative  to  achieve  a  high  degree 
of  technical  interoperability  to  minimize  risks  and  reconfiguration.  The  inset  diagram 
illustrates  the  basic  communications  topology. 

There  were  numerous  firewall, 
addressing  and  port  configuration 
issues  to  deal  with  at  each  endpoint. 

For  example,  certain  ports  had  to  be 
opened  on  the  respective  firewalls  to 
allow  specific  packets  to  pass,  in  and 
out.  This  spanned  the 
videoconferencing  and  kinematics 
control  data. 

Both  robots  utilized  the  same  video 
CODEC  (HaiVision  Hai  1060), 
network  protocols  and  bandwidth.  The 
HaiVision  is  an  MPEG4  AVC  H.264 
CODEC.  The  aquanauts  were  trained 
on  operation  of  the  HaiVision  CODEC 
but  once  connected  to  the  Aquarius  network  it  was  remotely  configured  and  tuned  by  Mr. 
Lance  Boulet  of  HaiVision  at  each  site. 

The  CODEC  encoded  video  stream  so  that  it  could  be  viewed  on  a  laptop  as  the  remote 
sites.  Although  picture  quality  was  excellent,  there  was  a  noticeable  latency  of  approximately 
500  -  1,000  millisecond.  This  was  due  to  compression  and  decompression  as  the  latency  of 
the  Internet  was  measured  at  about  70  ms. 

The  RAVEN  also  utilized  iChat  (V.2.1.3)  for  the  Apple  Macintosh. 

Technology  Summary 

The  challenges  of  implementing  the  technology  were  numerous.  Beyond  the  fact  that 
each  robot  is  experimental,  each  site  presented  unique  challenges.  As  mentioned  earlier,  at 
the  NURC  and  Aquarius  habitat,  the  challenges  were  mitigated  by  the  highly  interoperable 
design  efforts.  For  example,  the  mounting  superstructure  was  designed  to  support  both 
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robotic  platforms  even  though  the  M7  hung  from  an  overhead  perspective  and  the  RAVEN 
used  a  floor/table  mount. 

Network  configurations  were  much  more  challenging.  The  microwave  connection  was 
very  reliable  and  efficient  as  was  the  connection  to  the  Internet  through  TerraNova. 
However,  the  connections  in  Cincinnati  and  Nashville  were  difficult.  At  the  CMC,  outdated 
infrastructure,  old  devices  and  minimal  bandwidth  necessitated  creative  solutions. 

In  Nashville,  TATRC  and  UC  coordinated  access  to  a  6  Mbps  line.  This  was  delayed  due 
to  scheduling  constraints,  and  therefore,  impacted  pre-testing. 

Once  the  various  issues  were  resolved  at  all  locations,  the  interactions  took  place 
successfully. 

Project  Timeline 


Preparation  for  the  mission  occurred  in  the  months  prior  to  splash  down.  The  crew 
announcement  and  seminar  were  held  at  UC  in  March  2007.  Splash  down  (the  actual  diving 
to  Aquarius)  took  place  on  May  7,  2007.  This  mission  lasted  until  May  18,  2007,  when  the 
crew  splashed  up  (surfaced). 

Management  and  Organization 


The  UC  team  consisted  of  the  PI,  Mr.  Doam,  Mr.  Brett  Harnett,  Ms.  Elyssa  Westrich  and 
Dr.  Broderick.  During  the  months  preceding  the  mission,  the  UC  team  participated  in  MMT 
meetings,  which  were  held  weekly.  These  meetings,  held  by  teleconference,  were  also 
attended  by  a  variety  of  other  personnel,  the  NEEMO  crew  and  TATRC’s  Ms.  Nita 
Grimsley. 

The  organizations  and  their  representatives  involved  in  the  telesurgery  experiments 
included  UW  -  Mitch  Lum  and  Diana  Freeman;  HaiVision  -  Lance  Boulet;  SRI  -  Thomas 
Low  and  Kevin  Hufford;  CMAS  -  Dr.  Mehran  Anvari;  and  COL  Jon  Bowersox,  MC,  USAF- 
Ret  -  UC  and  Cincinnati  VA  Hospital.  Each  participant  also  included  a  number  of  other 
individuals  as  well. 

In  order  to  organize  the  use  of  the  Aquarius  Habitat  a  Memorandum  of  Understanding 
(MOU)  was  established  between  UC,  NASA  JSC  and  UNCW.  This  was  supplemented  with  a 
subaward  contract  between  UC  and  UNCW  for  the  operational  use  of  the  habitat. 

The  conduct  of  this  research  also  involved  a  requirement  for  having  research  reviewed  by 
the  Institutional  Review  Board  (IRB)  at  UC  and  the  U.S.  Army’s  HPRO  ORP. 
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Key  Research  Accomplishments 


The  conduct  of  the  NEEMO  12  mission  had  a  number  of  key  research  accomplishments. 

These  are  listed  below  in  two  categories  -  Primary  and  Secondary.  These  include  the 

following: 

Primary 

1.  Developed  and  conducted  research  on  the  world’s  first  semi-autonomous  ultrasound- 
guided  needle  insertion,  where  surgeon  and  simulated  blood  vessel  were  separated  by 
some  distance.  The  surgeon,  located  at  the  TATRC  exhibit  area  during  the  ATA 
meeting  in  Nashville,  used  the  M7  master  controller  to  position  the  ultrasound  probe 
remotely  and  then  the  robot  autonomously  inserted  a  needle  into  the  phantom  blood 
vessel. 

2.  Successfully  deployed  two  unique  surgical  robotic  systems  in  an  extreme 
environment  and  operated  them  remotely  over  a  wireless  communication  system. 

3.  The  NEEMO  12  crew  conducted  a  variety  of  research  activities  focused  on 
conducting  telesurgery  in  remote  and  extreme  environments. 

4.  Brought  together  a  world-class  team  of  telesurgery  experts  to  conduct  research, 
including  expertise  at  UC,  UW,  SRI  and  involving  Dr.  Bowersox  and  Dr.  Anvari. 

5.  Ensured  that  the  robotic  systems  were  easily  assembled  and  disassembled  such  that 
users  (as  opposed  to  the  developers  and  engineers)  could  deploy  them  in  an  extreme 
environment. 

6.  Began  to  integrate  and  these  systems  and  thereby  work  toward  the  goal  of  a  standard 
distributed  autonomous  therapeutic  robotic  platform/system. 

Secondary 

1.  Held  a  crew  announcement  at  the  University  of  Cincinnati  in  March  2007.  This  also 
provided  an  opportunity  for  a  special  panel  discussion  on  remote  healthcare.  In 
addition  to  the  crew,  the  panel  discuss  include  business  leaders,  engineers,  and 
medical  personnel.  TATRC  personnel,  Dr.  Christian  Macedonia  and  Ms.  Nita 
Grimsley  participated. 

2.  Conducted  a  robotics  competition  at  the  University  of  Cincinnati.  The  winners  of  the 
competitions  were  permitted  an  opportunity  to  drive  the  RAVEN  robot,  while  it  was 
deployed  in  the  habitat.  This  was  conducted  from  the  CMC. 

3.  UC  team  conducted  a  live  telesurgery  demonstration  to  TATRC  leadership  (COL 
Jonathan  Jaffin  and  COL  Ronald  Poropatich)  along  with  ATA  leadership  in  the 
TATRC  booth  at  the  ATA  meeting  in  Nashville,  TN. 

4.  Utilized  the  RAVEN  of  rock  manipulation.  Scientist  was  remotely  located  from  the 
robotic  system  and  rocks.  This  demonstrated  dual  use,  which  can  be  a  valuable  tool 
when  resources  are  constrained. 
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Reportable  Outcomes 


The  NEEMO  12  program  continues  to  receive  a  lot  of  attention  from  the  press,  and  there 
were  significant  outcomes  from  this  work.  The  work  accomplished  in  support  of  NEEMO  12 
has  been  reported  in  many  ways.  The  aforementioned  accomplishments,  most  notably  the 
semi-autonomous  needle  insertion,  have  been  and  will  be  reported  in  a  variety  of  venues. 
These  are  enumerated  below.  In  addition  to  this  final  report,  this  TATRC-funded  research 
will  be  published  and  presented.  There  has  been  significant  media  coverage  of  this  research. 
All  of  the  outcomes  are  reported  below.  Data  analysis  continues  and  TATRC  will  be 
acknowledged  in  all  peer-reviewed  publications  and  presentations. 

Intellectual  Property 

There  is  no  intellectual  property  to  report  from  this  research. 
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Locations.  World  Healthcare  Innovation  and  Technology  Congress  3.0  -  Innovation  to 
Transform.  Washington,  DC.  December  2007 
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Health  Seminar  that  International  Virtual  e-Hospital  Foundation,  Tirana  Albania,  October 
2007 

4.  Broderick,  TJ.  Surgery  in  Space.  1 6th  Society  of  Laparoendoscopic  Surgeons  Annual  Meeting 
and  Endo  Expo  2007,  San  Francisco,  CA.  September  2007 
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Publications 


1.  Doam  CR,  Hufford  K,  Low  T.  Rosen  J,  Hannaford  B.  Telesurgery  and  Robotics:  A 
Roundtable  Discussion.  Telemed  and E  Health  2007;  13(4):369-380. 

2.  M.  Lum,  et.  al.  Objective  Assessment  of  TeleSurgical  Robot  Systems:  Telerobotic  FLS. 
Proceedings  of  MMVR  2008,  Long  Beach,  CA,  January  2008.  At  Press 

3.  Sankaranarayanan  G,  Potter  L,  Hannaford  b.  Measurement  and  Simulation  of  Time  Varying 
Packet  Delay  with  Applications  to  Networked  Haptic  Virtual  Environments,'  Proceedings  of 
Robocom  2007,  Athens,  Greece,  October  2007. 

4.  Sankaranarayanan  G,  King  H,  Ko  SY,  Lum  MJH,  Friedman  D,  Rosen  J,  Hannaford  B. 
Portable  Surgery  Master  Station  for  Mobile  Robotic  Telesurgery.  Proceedings  of  Robocom 
2007,  Athens,  Greece,  October  2007. 

News  Coverage  (Television,  Radio,  Print  Media.  And  Web) 

UC 

March  2,  2007 
WKRC-TV,  Channel  12 

Recorded  pre-mission  story  about  NEEMO  12  featuring  Dr.  Timothy  Broderick.  Talking  about 
overall  objectives  of  the  mission.  Reported  by  Paul  Adler  during  the  6  a.m.  and  7  a.m.  morning 
broadcasts. 

March  6,  2007 

The  News  Record  (UC  student-run  newspaper) 

Brief  pre-mission  news  story  featuring  Dr.  Broderick,  detailing  NEEMO  12  science  objectives. 

March  6,  2007 
WXIX-TV/Fox  19 

Live  morning  show  appearance  by  Dr.  Broderick  and  Heidi  Stefanyshyn-Piper  on  local  Fox  19 
morning  show  to  talk  about  upcoming  mission  and  local  outreach  events. 

May  7,  2007 

WNKU-FM/89.7  FM— Northern  Kentucky  NPR 

Pre-recorded  interview  featuring  Mr.  Charles  Doam,  talking  about  the  upcoming  NEEMO  12  mission 
and  the  junior  high  robotics  competition,  hosted  by  UC. 

May  9,  2007 

WKRC-TV/CBS,  Channel  12 

Pre-recorded  segment  showing  undersea  NEEMO  12  team  talking  to  children  at  a  community 
outreach  event  at  the  Cincinnati  Museum  Center,  hosted  by  UC.  Reported  by  Liz  Bonis  at  5  p.m. 

June  1,  2007 

WKRC-TV/CBS,  Channel  12 

Pre-recorded  morning  show  interview  with  Dr.  Broderick,  on  NEEMO  12.  Reported  by  Paul  Adler 
during  the  6  a.m.  and  7  a.m.  broadcasts.  The  story  talked  about  results  of  the  mission  and  featured 
video  and  still  photos  from  when  the  team  was  undersea. 


19 


NASA  Extreme  Environment  Mission  Operations  (NEEMO)  12 


June  10,  2007 

WVXU-FM/91.7  FM— Greater  Cincinnati  NPR 

Pre-recorded  segment  featuring  Dr.  Broderick  and  his  recent  involvement  in  the  NEEMO  12  mission. 
Aired  during  “Cincinnati  Edition”  around  7:50  a.m. 

UC  PUBLICATIONS 

Findings,  April  2007 

Pre-mission  feature  story  detailing  UC  involvement  in  NASA  NEEMO  12  mission  and  science 
objectives.  Newsletter  has  a  wide  distribution  to  faculty,  staff  and  friend  of  UC  as  well  as  university 
clinical  practice  areas. 

Findings,  May  2007 

Pre-mission  photo  story  about  local  robotics  competition  tied  to  NEEMO  12  mission  and  science 
objectives. 

Findings,  June  2007 

Photo  story  showing  local  student  remotely  “driving”  the  surgical  robot  from  the  Cincinnati  Museum 
Center  during  the  NEEMO  12  undersea  mission. 

OTHER 

Dr.  Broderick  was  also  interviewed  on  Tuesday,  July  3  at  1:30  p.m.  about  telemedicine  and  robotics 
for  a  show  that  will  air  on  National  Geographic  and  the  Armed  Forces  Network. 

SRI,  International 

SRI  has  a  number  of  citations  of  news  reports.  These  are  highlighted  in  their  report.  See 
Appendix  C. 

UW  -  BRL 


UW  has  a  number  of  citations  of  news  reports.  These  are  highlighted  in  their  report.  See 
Appendix  D. 

Military  Relevance 


Provision  of  emergency  diagnostic  and  medical  care  in  an  extreme  environment  is  of 
significant  interest  to  the  U.S.  Army.  This  research  represents  an  important  step  in  the 
evolution  of  surgical  robotics  from  telesurgery  to  distributed  autonomous  therapeutics  that 
includes  remote  supervisory-controlled,  semi-autonomous  robotic  function.  This  research 
marked  the  first  time  ever  that  a  surgeon,  remotely  located  from  the  robotic  system  and 
simulated  tissue,  was  able  to  insert  a  needle  into  a  blood  vessel  using  ultrasound-image 
guidance.  More  importantly,  the  robotic  system  was  able  to  autonomously  insert  a  needle 
into  a  simulated  blood  vessel.  Further  distributed  autonomous  therapeutics  research  is 
indicated  Robotic  telesurgery  and  autonomy  have  been  identified  by  TATRC  as  a  prime  area 
of  medical  research  for  the  next  decade.  These  technologies  represent  key  components  of 
DARPA’s  and  the  U.S.  Army’s  Trauma  Pod  Program.  NEEMO  12  research  has  been  a 
natural  progression  from  the  work  conducted  in  NEEMO  7  and  NEEMO  9. 
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NEEMO  12  has  also  provided  a  foundation  for  leverage  of  diverse  experience  and 
resources  of  multiple  branches  of  the  U.S.  military,  which  has  been  characterized  in  prior 
NEEMO  missions.  Collaboration  across  traditional  government  barriers  has  accelerated 
medical  technology  development  as  demonstrated  by  multiple  NEEMO  scientific  and 
popular  presentations/publications.  NEEMO  12  built  upon  prior  collaboration  and  success  to 
catalyze  further  development  of  key  TATRC  relationships  and  technologies. 

Budget 


The  value  of  this  award  was  $244,543.  An  apportionment  of  $75K  was  sent  directly  to 
NASA  JSC  through  an  interagency  transfer.  The  UC  grant  was  $169,543.  A  subaward  in  the 
amount  of  $110,000  was  established  with  UNCW  for  use  of  the  Aquarius  Habitat.  The 
remainder  of  the  funds  was  used  to  support  salary,  fringe,  travel  and  equipment  at  UC.  Dr. 
Broderick’s  participation  was  part  of  his  assigned  task  as  an  IP  A  with  TATRC.  The  funds 
were  spent  in  accordance  with  the  budget  as  it  was  submitted. 

The  subaward  contracts  with  SRI  and  UW  were  accomplished  through  the  TATRC- 
funded  Advanced  Center  for  Telemedicine  and  Surgical  Innovation  (ACTSI)  grant. 

Conclusions 


TATRC-led  robotic  surgery  development  has  continued  with  this  research.  During  this 
mission,  two  surgical  robotic  systems,  the  SRI  M7  and  the  UW  RAVEN  were  modified  and 
deployed  in  an  extreme  environment,  the  Aquarius  Habitat.  This  habitat  is  an  ideal  research 
platform  to  spur  collaborative  accelerated  development  of  advanced  medical  technology  for 
use  in  extreme  environments.  Both  systems  were  successfully  deployed,  assembled  and 
dissembled  by  trained  personnel,  and  the  systems  were  shown  to  be  effectively  controlled 
from  remote  sites. 

The  robotic  telesurgery  research  brought  together  academia,  industry,  and  government  to 
expand  understanding  of  robotic  telesurgery  in  extreme  environments.  The  research  partners 
included  NOAA/UNCW  (telecommunication),  TATRC,  UW,  SRI,  and  HaiVision. 

This  funded  research  had  several  key  activities  (1)  evaluate  the  ability  to  perform  a  semi 
autonomous  task,  (2)  deployment  of  two  different  surgical  robots  in  the  extreme  environment 
and  evaluate  their  performance;  and  (3)  build  upon  relationships  to  move  telesurgery 
forward. 

During  this  research,  both  robotic  systems  were  operated  remotely.  This  was 
accomplished  using  a  robust  and  reliable  telecommunications  link  between  the  habitat  and 
several  other  locations.  A  link  was  established  between  the  habitat  and  the  convention  center 
in  Nashville  to  support  the  TATRC  exhibit  at  that  ATA  annual  meeting.  During  this  link, 
TATRC  leadership  discussed  telesurgery  with  Drs.  Anvari  and  Bowersox  and  Mr.  Doarn. 
The  assembled  group  interacted  with  the  NEEMO  crew.  A  surgeon  in  Nashville  conducted 
the  first  semi-autonomous,  ultrasound  image-guided  needle  placement  in  a  simulated  blood 
vessel  using  the  M7. 
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A  second  set  of  telesurgery  tasks  that  were  conducted  using  the  UW  RAVEN.  These 
tasks  included  remote  surgeons  conducting  basics  surgical  tasks  on  the  SAGES  FLS  training 
system.  The  RAVEN  was  also  controlled  by  several  individuals  remotely  located.  These 
individuals  manipulated  various  simulated  structures  in  the  habitat  over  the 
telecommunications  link.  The  users  were  located  in  Cincinnati,  Key  Largo,  Nashville,  and 
Seattle. 

Both  systems  demonstrated  the  ability  to  be  deployed  in  an  extreme  environment  by 
novice  users  and  operated  from  a  distant  site.  A  key  milestone  was  the  remote  semi- 
autonomous  insertion  of  a  needle  in  a  simulated  blood  vessel  using  a  scalable 
telecommunications  link  and  a  modified  telerobotic  surgery  system  (M7).  This  research 
suggests  that  evolution  from  telesurgery  to  distributed  autonomous  therapeutics  can 
overcome  operational  communication  issues  routinely  encountered  during  military  conflict. 

Provision  of  emergency  diagnostic  and  medical  care  in  an  extreme  environment  is  of 
significant  interest  to  the  U.S.  Army.  This  project  substantially  furthered  military  robotic 
surgical  research  and  represents  an  important  step  in  our  march  toward  a  deployable  robotic 
system  that  improves  the  access  to  and  quality  of  care  on  the  battlefield. 
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Appendix  A 
Contact  Information 


Principal  Investigator: 

Charles  R.  Doarn,  MBA 

Executive  Director 

Center  for  Surgical  Innovation  (CSI) 

Deputy  Director,  Advanced  Center  for  Telemedicine  and  Surgical  Innovation  (ACTSI) 

Associate  Professor  of  Surgery  and  Biomedical  Engineering 

Editor-in-Chief 

Telemedicine  and  E-Health  Journal 
Administrative  Director 

Minimally  Invasive  Medical  Technology  Center  (MIMTeC)  an  NSF  Center 

Department  of  Surgery 

University  of  Cincinnati 

231  Albert  Sabin  Way,  MSB  1466 

ML  0558 

Cincinnati,  OH  45267-0558 
Voice:  (513)558-6148 
Fax:(513)558-2026 
E-mail:  charles.doarn@uc.edu 
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Appendix  B 

Acronyms 

ACTSI 

ATA 

Advanced  Center  for  Telemedicine  and  Surgical  Innovation 
American  Telemedicine  Association 

BMIST 

BRL 

Battlefield  Medical  Infonnation  System-Tactical 
BioRobotics  Laboratory 

CODEC 

CMAS 

CMC 

CMO 

CSI 

Coder  /  Decoder 

Center  for  Minimal  Access  Surgery 

Cincinnati  Childrens  Museum 

Chief  Medical  Officer 

Center  for  Surgical  Innovation 

DARPA 

Defense  Advanced  Research  Projects  Agency 

ELAN 

Extended  Local  Area  Network 

FLS 

Fundamentals  of  Laparoscopic  Surgery 

HAPsMRT 

HPRO 

High  Altitude  Platforms  for  Mobile  Robotic  Telesurgery 
Human  Protection  Research  Office 

IP 

IRB 

ISP 

Internet  Protocol 

Institutional  Review  Board 

Internet  Service  Provider 

Kbps 

Kilo  bits  per  second 

LAN 

LSB 

Local  Area  Network 

Life  Support  Buoy 

Mbps 

MMT 

MOU 

ms 

Mega  bits  per  second 

Mission  Management  Team 

Memorandum  of  Understanding 
millisecond 

NASA 

NEEMO 

NO  A  A 

NSBRI 

NURC 

National  Aeronautics  and  Space  Administration 

NASA  Extreme  Environments  Mission  Operations 

National  Oceanographic  and  Atmospheric  Administration 
National  Space  Biomedical  Research  Center 

National  Undersea  Research  Center 
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OPR 

Office  of  Research  Protection 

SAGES 

SRI 

Society  of  American  Gastrointestinal  Endoscopic  Surgeons 
Stanford  Research  Institute 

TATRC 

TCP/IP 

Telemedicine  and  Advanced  Technology  Research  Center 
Transfer  Control  Protocol/Intemet  Protocol 

UAV 

UC 

UNCW 

USAMRMC 

UW 

Unmanned  Airborne  Vehicle 

University  of  Cincinnati 

University  of  North  Carolina  at  Wilmington 

United  States  Army  Medical  Research  and  Materiel  Command 
University  of  Washington 
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Appendix  C 
SRI  Report 

The  appended  report  is  as  it  was  prepared  by  the  SRI.  It  has  been  formatted  to  fit  this  final 

report. 
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NEEMO  12  Report 
SRI  International 

Prepared  By: 

Kevin  Hufford 
Thomas  Low 
SRI  International 
12/04/07 
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Purpose 

The  NEEMO  12  teleoperation  experiments  demonstrated  the  ability  of  a  robotic  system 
to  mitigate  the  effects  of  network  latency  through  the  conduct  of  autonomous  robotic 
subtasks.  It  demonstrated  the  feasibility  of  integrating  interoperative  imaging  with  needle- 
based  robotic  surgical  procedures,  an  important  milestone  for  the  use  of  robotics  for  trauma 
care.  The  capability  of  a  robotic  system  to  perfonn  surgical  subprocedures  autonomously  is 
an  important  step  toward  mitigating  the  impact  of  communication  latency  caused  by  long 
distance  teleoperation. 

The  use  of  telerobotics  for  delivery  of  life-saving  procedures  in  the  battlefield,  in  space, 
or  during  patient  evacuation  is  of  interest  to  both  NASA  and  the  military.  Presently,  there  is  a 
shortage  of  critical  care  air  transport  teams;  telerobotic  capabilities  could  serve  as  a  force 
multiplier  to  enable  teams  to  transport  more  patients  and  allow  transport  with  less  skilled 
teams.  The  ability  to  perfonn  simple  procedures  could  allow  less  skilled  providers  to  care  for 
increasingly  ill  patients.  Automated  telerobotic  intensive  care  unit  services  provided  remotely 
by  an  intensivist  and  supported  in  air  by  a  technician  will  both  improve  care  and  reduce  the 
logistical  footprint  of  the  process. 

Goals 

The  NEEMO  analogue  missions  are  the  execution  of  space  processes  within  the  Aquarius 
underwater  laboratory  in  Key  Largo,  Florida.  Performing  innovative  medical  procedures  in 
this  remote  environment  allows  astronauts  and  their  support  crews  to  evaluate  and  train  for 
the  adoption  of  such  techniques  within  extreme  conditions  of  space  missions. 

SRI’s  participation  in  NEEMO  12  focused  on  demonstrating  a  combined  teleoperated  and 
autonomous  robotic  medical  task,  using  both  stereo  video  and  ultrasonic  imaging  to  locate 
and  insert  a  needle  into  fluid-filled  tube  contained  within  a  tissue  phantom.  The  primary 
objective  was  to  evaluate  the  use  of  autonomous  robot  function  as  a  means  to  overcome 
latency.  Further,  the  experiments  demonstrated  the  ability  of  non-specialists  in  an  extreme 
environment  to  rapidly  deploy  a  two-armed  teleoperation  system  with  stereo  vision  and 
diagnostic  and  interventional  imaging  capabilities. 

Accomplishments 

The  goal  of  this  experiment  was  fully  met.  The  master  control  console  was  set  up  on  the 
show  floor  of  the  ATA  2007  conference  in  Nashville,  TN.  The  slave  robot  was  deployed  in 
the  Aquarius  habitat  off  the  coast  of  Key  Largo,  FL.  A  live  demonstration  of  the  remote 
image-guided  needle  insertion  was  successfully  performed  for  attendees  of  the  ATA 
conference. 
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Figure  1:  M7  Master  console 


Figure  2:  M7  Robot  as  deployed  in  Aquarius  habitat  (Photo  credit:  NASA) 
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Figure  3:  M7  Robot  holding  ultrasound  probe  and  needle 


Workflow 

A  remote  image-guided  needle  insertion  follows  the  sequence  below. 

1 .  Surgeon  places  ultrasound  probe  on  vessel  phantom  (Figure  4). 

2.  Surgeon  orients  ultrasound  probe  while  viewing  image.  Pressure  applied  to  phantom 
is  maintained  by  robot.  Position  and  attitude  of  probe  is  adjusted  through 
teleoperation  (Figure  5). 

3.  Surgeon  positions  needle  to  point  on  desired  insertion  path  using  stereo  view  (Figure 

6). 

4.  Surgeon  uses  master  console  to  view  ultrasound  image  and  to  select  target  on 
ultrasound  image.  Tip  of  needle  is  held  fixed  during  target  selection,  but  needle 
orientation  is  continuously  updated  to  align  with  target.  (Figure  7). 

5.  When  commanded,  robot  autonomously  inserts  needle  along  its  axis  to  target  point 
depth  (Figure  8 -Figure  10).  Surgeon  views  in  3-D. 

6.  When  commanded,  robot  retracts  needle  out  of  vessel  phantom  and  returns  control  to 
surgeon  (Figure  11). 
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Figure  4:  Surgeon  positions  probe  on  phantom 


Figure  6:  Surgeon  positions  needle  at  insertion 
point 


Figure  8:  Ultrasound  before  needle  insertion 


Figure  9:  Robot  inserts  needle  to  target 


Figure  5:  Surgeon  orients  probe  for  best  image 


Figure  7:  Surgeon  moves  target  cursor  on 
ultrasound  image  and  selects  insertion  target 
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Budget 

At  the  initiation  of  the  project,  the  M7  robotic  system  was  a  functional  telesurgery 
system,  but  the  software  architecture  at  the  time  did  not  allow  for  significant  development 
beyond  the  current  capabilities  of  the  system.  At  the  initiation  of  the  project,  the  system 
functioned  as  follows: 

•  Similar  Master  and  Slave  kinematics  required:  The  master  and  slave  robotic 
systems  were  kinematically  similar,  requiring  a  large  master  console,  and  restricting 
control  of  the  slave  robot  to  a  single  master  console. 

•  Joint  to  joint  control:  Due  to  the  kinematic  similarity  of  master  and  slave,  the 
system  functioned  by  transmitting  the  encoder  values  of  each  individual  joint  of  the 
master  control  arms  to  the  corresponding  joint  on  the  slave  robot  arms. 

•  No  clutching:  As  a  result  of  the  joint-to-joint  control,  it  was  not  possible  to  allow  the 
surgeon  to  clutch,  or  disengage  control  momentarily,  to  allow  him  to  reposition  his 
hands  to  a  more  comfortable  working  pose. 

•  No  Scaling:  Due  to  the  kinematically-similar  master  and  slave,  it  was  possible  for  the 
surgeon  to  reach  nearly  all  of  the  workspace  of  the  slave  robotic  system.  However,  it 
was  not  possible  to  scale  the  surgeon’s  motions  down  to  allow  better  performance  in 
delicate  operations  requiring  high-fidelity  small-scale  motion. 

•  No  Autonomous  Action:  The  slave  follows  the  master,  but  could  not  be  commanded 
to  move  in  a  prescriptive  manner. 

To  enable  the  robotic  demonstration  for  NEEMO  12,  SRI  developed: 

•  Forward  and  inverse  kinematics  iterative  solution:  Prior  SRI  IR&D  funded 
development  of  closed-form  forward  and  iterative  inverse  kinematic  solution  was 
leveraged  to  permit  conversion  between  real-world  Cartesian  coordinates  and  robot 
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joint  encoder  values.  This  capability  was  vital,  for  it  provided  the  ability  to  issue 
autonomous  position  commands  as  demonstrated  during  NEEMO  12,  as  well  as 
pennitting  the  use  of  a  smaller,  kinematically-dissimilar  master  controller,  as 
demonstrated  in  experiments  since  NEEMO  12. 

•  Graphical  Simulation  (funded  by  SRI  IR&D):  A  3D  graphical  simulation  of  the 
robot  kinematics,  developed  on  SRI  internal  funds,  allowed  new  algorithms  to  be 
tested  as  a  safety  precaution  before  committing  the  M7  robot  hardware.  This 
simulation  also  pennitted  simultaneous  coding  by  multiple  developers  without  being 
limited  to  sharing  a  single  piece  of  hardware. 

•  New  communication  protocol  (incremental  moves)  packet:  A  new  command 
packet  structure  from  the  master  to  the  slave  was  created  that  incorporated 
incremental  motion  commands.  These  incremental  position  commands  prevent  large 
jerks  in  position  that  could  otherwise  be  seen  if  an  intennediate  absolute  position 
command  was  lost. 

•  Restructured  control  and  communication  software:  SRI  redesigned  the  packet 
structure  used  for  robotic  control  to  reduce  the  network  bandwidth  usage  and  made 
other  code  improvements  to  perform  more  reliably  in  unpredictable  network 
situations.  The  system  was  designed  to  hold  its  last  position  in  the  event  of 
communications  interruption. 

•  Slave  status  packets  containing  achieved  position  data:  Status  packets,  otherwise 
known  as  “heartbeats”,  sent  back  multiple  times  per  second  from  the  slave  robotic 
system,  were  implemented  and  contain  the  current  position  data,  which  was  then  be 
used  to  verify  that  the  commanded  position  was  achieved.  If  communication  is  lost 
between  master  and  slave,  the  system  remains  in  a  safe  state.  When  communication  is 
restored,  the  operation  resumes  seamlessly. 

•  Clutching:  SRI  modified  the  software  to  enable  scaling  of  the  linear  motion  between 
master  and  slave  and  to  enable  the  surgeon  to  clutch  (temporarily  disengage 
master/slave  control)  to  allow  repositioning  to  a  more  comfortable  user  pose. 

Software  and  Hardware  improvements: 

•  Calibration  Technique  for  determination  of  world  to  arm  coordinate 
transformation.  To  pennit  each  arm  to  be  commanded  to  a  pose  is  space,  one  must 
first  know  precisely  the  position  and  orientation  of  each  ann’s  base  in  the  world 
coordinate  frame.  SRI  developed  a  software  program  to  detennine  these  reference 
frames  experimentally.  A  straight  edge  is  placed  on  the  work  surface,  aligned  with 
first  the  X  axis,  and  then  the  Y  axis.  These  axes  were  projected  onto  the  work  surface 
with  a  laser  cross  hair  projector.  A  stylus-like  tool  in  inserted  in  the  tool  holder,  and 
each  ann  manually  moved  along  the  straight  edge.  At  points  along  each  axis  are 
digitized,  and  a  least  squares  3-D  line  fitted  to  the  cloud  of  points.  These  two  lines  for 
each  arm  detennine  the  arms  X  and  Y  axis.  The  Z  is  obtained  by  taking  the  cross 
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product  of  these  two  vectors.  The  base  transformation  matrix  that  locates  each  ann  is 
determined  by  constraining  the  X  and  Y  axes  of  each  arm  to  correspond  in  space. 

•  Trapezoidal  velocity  motion  profile:  To  provide  for  a  smooth  motion  profile  for 
needle  insertion,  a  trapezoidal  velocity  motion  profile  was  used  when  planning  the 
automated  move.  Interpolation  was  perfonned  in  world  space.  The  linear  path  of  the 
move  was  divided  into  non-unifonn-length  steps  to  provide  a  smoother  start  and  stop 
to  the  needle  insertion. 

•  User  Interface  Improvements:  SRI  developed  a  GUI  to  permit  untrained  operators 
to  use  the  system.  This  entailed  digitization  of  three  video  streams  from  the 
ultrasound  instrument  and  from  the  left  and  right  eye  cameras.  A  video  capture  card 
(Eurosys  Piccolo  Tetra)  capable  of  four  channels  of  NTSC  video  capture  was 
installed  in  a  PCI  express  buss  industrial  PC.  Depending  on  the  step  of  the  procedure, 
either  the  ultrasound  video  or  stereo  camera  video  was  presented  in  the  surgeon 
console.  Overlaid  on  the  lower  part  of  the  image  were  step-by-step  procedure 
instructions,  indicating  the  present  task,  the  next  task  (to  the  right)  and  the  previous 
task  (to  the  left).  These  transparent  overlays  were  generated  using  an  Nvidia  7800 
GTX  graphics  processor  in  the  industrial  PC  case.  Each  CRT  monitor  in  the  master 
console  was  connected  directly  to  the  graphics  card  output  using  special  cables  to 
combine  horizontal  and  vertical  sync  into  a  composite  sync  signal.  Foot  pedals  were 
used  to  permit  the  operator  to  advance  to  the  next  step,  or  when  appropriate,  to  revert 
to  the  previous  step.  A  third  foot  pedal  was  used  to  clutch  or  disconnect  the  slave  fro 
the  master.  When  pressed,  master  control  movement  was  not  transmitted  to  the  slave, 
but  packets  with  zero  movement  increment  were  transmitted  to  prevent  the  slave  from 
going  into  a  loss  of  signal  safe  mode. 

•  Hardware  adapter  to  hold  needle  and  probe:  To  enable  the  experiment,  it  was 
necessary  to  design  adapters  for  both  the  needle  and  ultrasound  probe.  These  adapters 
connected  the  universal  tool  interface  at  the  end  of  the  robot  arms  with  the  needle  and 
ultrasound  probe,  respectively.  In  addition,  the  necessary  kinematic  changes  were 
made  in  software  to  provide  correct  robot  motion  depending  on  the  tool  that  was 
inserted. 

•  Structure  modifications  to  support  structure  in  the  habitat:  Partnering  with  the 
University  of  Washington  researchers,  a  support  structure  was  designed  to  mount  the 
M7  slave  robot  in  the  Aquarius  habitat.  This  extruded  aluminum  structure  was 
required  to  be  easily  deployable  and  to  mount  on  the  edges  of  the  hunks  in  the  habitat. 
The  structure  was  designed  so  that  the  vast  majority  of  it  would  be  useful  for  both  the 
M7  and  Raven  robots. 

•  Servo  rates  on  PMAC  raised:  To  provide  smoother  motion  and  a  more  tightly- 
controlled  servo  system,  the  servo  loop  closure  rates  of  the  servo  motion  controller 
cards  for  the  M7  robot  anns  (called  PMACs)  were  raised  to  2  kHz.  These  PMAC 
cards  provide  high-speed  low-level  servo  loop  closure,  while  the  commands 
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transmitted  from  the  master  to  the  slave  were  sent  at  a  lower  rate  -  approximately  200 
Hz  during  the  NEEMO  12  experiments. 

•  Support  for  autonomous  needle  insertion:  Software  was  created  to  perform 
mapping  between  the  2D  target  position  on  the  ultrasound  image  and  the  location  of 
that  target  point  in  3D  space.  More  details  on  the  overall  software  architecture  may  be 
found  below  under  the  Autonomy  Overview 


NEEMO  12  Mission  Support:  SRI  staff  members  prepared  the  M7  equipment  for 
shipment  to  and  from  the  Nashville,  TN  and  Key  Largo,  FL  sites  and  were  present  onsite  in 
Nashville,  TN  during  performance  of  the  experiments. 

Autonomy  Overview 

During  the  initial  portion  of  the  needle-insertion  procedure,  the  slave  robot  was 
manipulated  by  the  surgeon  sitting  at  the  master  console  in  nonnal  teleoperated  mode.  In  this 
mode,  as  the  surgeon  moved  the  arms  of  the  master  console,  the  differential  position  packets 
in  Cartesian  coordinates  were  transmitted  to  the  slave  robot,  whose  arms  then  moved  to  the 
commanded  positions. 

During  subsequent  steps  of  the  procedure,  some  of  the  directions  of  motion  were 
restricted  for  safety  and  ease  of  use  of  the  system  by  the  surgeon.  Upon  transition  into  the 
autonomous  portion  of  the  procedure,  the  surgeon’s  only  inputs  were  two  pedals — one  of 
which  advanced  the  system  through  the  steps  of  the  procedure,  and  the  other  which  reverted 
the  system  to  the  previous  step. 

After  the  autonomous  removal  of  the  needle,  the  next  state  returns  nonnal  teleoperation 
control  to  the  surgeon.  Figure  12  shows  a  brief  state  machine  diagram  illustrating  transitions 
between  autonomous  and  teleoperated  control. 
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DeltaPositionPacketReceived  /ignored 

Figure  12:  Control  moves  between  surgeon  teleoperation  and  autonomous  control  depending  on  the  next 
logical  state  for  a  needle  insertion  procedure. 

State  Machine 

Three  foot  pedals  allowed  the  surgeon  to  clutch,  move  through  the  procedure,  and  revert 
to  the  previous  step  in  the  procedure,  respectively.  However,  since  returning  to  a  previous 
state  would  have  resulted  in  an  unsafe  situation  during  certain  portions  of  the  procedure,  all 
state  machine  transitions  were  verified  to  move  to  safe,  logical  conditions  based  upon  the 
current  state.  For  instance,  after  autonomous  needle  insertion,  simply  returning  to  the 
“position  needle”  phase  could  result  in  undesired  motion  of  the  needle  inside  the  tissue.  Thus, 
as  a  safeguard,  pressing  on  the  “back”  pedal  after  an  autonomous  needle  insertion  resulted  in 
an  autonomous  needle  removal  along  the  needle  axis  to  a  safe  position  above  the  tissue.  Once 
the  needle  had  been  fully  retracted,  teleoperation  control  was  returned  to  the  surgeon  for 
needle  re -positioning. 

Constraints  of  motion  depending  on  current  state 

Depending  on  the  current  step  in  the  sequence,  the  permitted  directions  of  motion  of  the 
robot  were  constrained  for  reasons  of  safety  and  ease  of  use  by  the  surgeon.  For  instance, 
during  placement  of  the  ultrasound  probe  on  the  vessel  phantom,  the  surgeon  interacts  with 
the  system  using  a  3-D  stereoscopic  view.  Moving  into  the  “orient  probe”  step  changes  the 
view  presented  to  a  2-D  ultrasound  image,  and  constrained  the  vertical  motion  of  the 
ultrasound  probe,  allowing  the  surgeon  to  slide  the  probe  across  the  surface,  and  adjust  its 
orientation,  while  maintaining  constant  contact  pressure.  Movement  of  the  master 
commanding  slave  motion  normal  to  the  phantom  surface  plane  were  ignored.  This  prevented 
the  surgeon  from  over  compressing  the  tissue  with  the  probe,  and  thus  distorting  the  vessel 
shape. 

Once  the  probe  had  been  placed  and  oriented  correctly,  moving  into  the  “position  needle” 
phase  locked  the  position  of  the  ultrasound  probe,  creating  a  reliable,  fixed  image  upon 
which  to  plan  the  needle  insertion. 
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Needle  insertion  planning 

For  planning  the  needle  insertion,  software  was  created  to  map  between  the  selected  2D 
target  position  on  the  ultrasound  image  and  the  location  of  that  target  point  in  3D  space. 

After  placing  the  ultrasound  probe  and  moving  the  needle  to  the  starting  point  of  the 
insertion  path,  the  surgeon  graphically  selected  the  target  point  on  the  ultrasound  image  by 
using  the  hand  controller.  During  this  portion  of  the  procedure,  only  a  2-D  ultrasound  view 
with  a  cross-hair  cursor  is  visible  to  the  surgeon.  Hand  controller  position  inputs  are  ignored 
and  the  position  of  the  needle  tip  is  maintained.  Rotation  of  the  hand  controller 
simultaneously  adjusted  both  the  orientation  of  the  needle,  and  the  position  of  the  displayed 
cross-hair  on  the  2-D  image.  The  fixed  needle  tip  position  avoids  potential  damage  or 
accidental  penetration  of  the  phantom  with  the  needle  during  the  “blind”  manipulation.  The 
displayed  cross-hair  position  was  calculated  to  correspond  to  the  3-D  intersection  of  the 
vector  defined  by  the  needle  orientation  and  tip  coordinate,  and  the  ultrasound  image  plane. 

Once  the  target  point  was  selected  from  the  2-D  ultrasound,  the  linear  needle  insertion 
distance  was  calculated.  To  avoid  penetrating  ancillary  tissue,  this  needle  insertion  depth  was 
verified  to  not  exceed  a  maximum  insertion  depth.  The  foot  pedal  initiated  command  from 
the  surgeon  initiates  the  needle  insertion  along  the  verified  path  and  depth  to  the  target 
position.  A  5-second  delay  was  added  to  permit  cancellation  of  the  insert  command,  should 
the  surgeon  see  fit.  A  trapezoidal  velocity  profile  was  used  during  the  linear  path  move  to 
insure  smoothness. 

Data 

For  the  NEEMO  12  experiments,  the  master  console  was  set  up  in  Nashville,  Tennessee, 
and  the  slave  robot  was  deployed  into  the  Aquarius  habitat  by  the  NEEMO  12  crew.  The 
remote  image-guided  needle  insertion  was  successfully  performed  multiple  times  between 
May  11  and  May  13,  2007.  The  experiments  culminated  in  a  successful  live  demonstration 
for  attendees  of  the  ATA  2007  conference. 

During  the  experiment,  a  3Mbps  Internet  connection  was  used  to  connect  the  master 
console  from  the  show  floor  of  the  ATA  conference.  On  the  Haivision  encoder,  each  of  the 
three  video  streams  (left  and  right  stereo  camera  views,  ultrasound)  was  set  to  500  Kbps,  and 
the  robot  control  data  used  approximately  500  Kbps.  During  the  experiment,  network 
latencies  between  Nashville  and  the  Aquarius  habitat  ranged  from  60-120  ms. 

Problems  with  technology 

Insertion  into  initialization  fixture:  For  robot  initialization  and  calibration  of  its 
encoders,  the  robot  arms  must  be  manually  placed  into  an  initialization  fixture,  which  mates 
with  a  round  metal  collar  on  each  robot  arm.  A  locking  pin  will  only  insert  correctly  in  one 
orientation  of  the  robot  arm,  but  there  is  nothing  to  prevent  the  arm  from  being  inserted  in 
another,  incorrect  orientation.  During  performance  of  some  of  the  experiments,  it  was  not 
apparent  that  the  robot  arm  had  been  improperly  inserted  into  the  initialization  fixture.  When 
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the  surgeon  began  to  use  the  robot,  the  directions  of  motion  were  mapped  incorrectly.  Upon 
reinitialization  with  proper  arm  insertion,  the  robot  worked  correctly.  To  address  this 
potential  use  error,  the  initialization  fixture  and  the  round  metal  collar  on  the  robot  will 
modified  to  an  asymmetrical  shape  that  will  prevent  improper  insertion. 

Removal  from  initialization  fixture:  Upon  startup,  with  direct  teleoperation,  it  is 
somewhat  difficult  for  the  surgeon  to  move  the  slave  robot  arms  out  of  the  initialization 
fixtures,  since  a  slight  twisting  in  orientation  can  cause  interference  between  the  initialization 
fixture  and  the  mating  collar  on  the  robot  arm.  During  conduct  of  the  final  experiment,  this 
interference  caused  the  robot  arm  to  become  stuck  in  the  initialization  fixture,  building  up 
tension  in  the  robot  cables.  The  robot  was  restarted,  thus  relaxing  and  then  re-tensioning  the 
cables.  When  the  robot  began  to  move,  one  of  the  cables  slipped  off  a  pulley,  and  the 
NEEMO  12  crew  was  unable  to  reseat  the  cable  in  the  confined  Aquarius  habitat. 

Since  the  NEEMO  12  experiments,  the  software  has  been  modified  to  minimize  dramatic 
changes  in  cable  tension  on  startup.  More  significantly,  autonomous  removal  of  the  robot 
arms  from  the  initialization  fixtures  has  been  implemented.  This  autonomous  motion  moves 
the  slave  robot  arms  from  the  initialization  fixtures  into  the  surgeon’s  field  of  view  upon 
startup.  Once  the  autonomous  motion  is  complete,  the  surgeon  is  free  to  control  the  robot. 

Presentations 

A  live  demonstration  of  the  remote  image-guided  needle  insertion  was  successfully 
performed  for  attendees  of  the  ATA  2007  conference  in  Nashville,  Tennessee  on  May  13, 
2007.  The  master  console  in  Nashville  was  used  to  control  the  robot  deployed  in  the 
Aquarius  habitat  off  the  coast  of  Key  Largo,  Florida. 

NEEMO  12  Media  Coverage 

SRI’s  participation  in  NEEMO  12  was  covered  by  a  number  of  media  channels: 

San  Jose  Mercury  News:  Robo-MD:  Menlo  Park’s  SRI  International  Designs  Surgical  Devices  for 
Hospitals,  Space,  Battlefield 

This  article  about  telesurgery  focuses  on  recent  advances  and  milestones  in  the  field.  The  article 
describes  how  NASA  will  conduct  its  second  test  in  a  year  of  a  robot  made  by  SRI  to  determine  its 
feasibility  for  use  on  a  future  mission.  SRI’s  Tom  Low,  director  of  the  medical  systems  and  robotics 
program,  is  quoted  in  the  article. 

URL:  http://www.mercurynews.com/business/ci_5831343 
Also  See: 

Akron  Beacon  Journal/Ohio,  com 

http://www.ohio.com/mld/ohio/news/nation/17200212.htjn 

Contra  Costa  Times 

http://www.contracostatimes.com/business/ci  5844746 
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Real  Cities 

http://www.realcities.  com/mld/realcities/news/nation/1 7200212.htm 

Red  Orbit 

http://www.redorbit.com/news/technology/926501/robomd_surgical_devices_for_hospitals_space_ba 

ttlefield/index.html 

Space  Ref.com:  NASA  Space  Simulation  and  Training  Project:  NEEMO  12 

This  announcement  is  a  status  report  from  the  12th  Mission  of  NASA  Extreme  Environment  Mission 
Operations  (NEEMO).  The  announcement  describes  the  crew  and  experiments  that  will  take  place. 

SRI’s  surgical  robot  is  noted  as  one  of  the  telesurgery  demonstrations. 

URL :  http  ://www.  spaceref.  com/news/viewsr.html?pid=24 1 06 

USA  Today:  Aquanauts  test  robotic  surgeons  in  undersea  lab 

This  article  is  about  NASA’s  Extreme  Environment  Mission  Operations  (NEEMO)  12  expedition. 

The  article  explains  that  in  the  mission,  a  team  of  six  aquanauts  and  two  robotic  surgeons  will  plunge 
into  the  Atlantic  Ocean  to  test  new  medical  and  exploration  tools  for  long  duration  spaceflight.  The 
article  notes  that  SRI’s  M7  surgical  robot  will  be  tested  on  the  NEEMO  12  mission. 

URL:  http://www.usatoday.com/tech/news/robotics/2007-05-07-robot-undersea-test_N.htm 

Also  See: 

Yahoo!  News 

http://news.vahoo.eom/s/space/20070507/sc  space/deepdivinzteamtotestroboticsurseonsonseafloor 

Fox  News 

h  ttp.Y/www.foxnews.  com/story/ 0%2  C2 933%2  C27053 0%2  COO.h  tm  l 

Robotics  Online :  Robot  Surgery:  Get  the  Scoop  at  the  International  Robots  &  Vision  Show 

This  brief  article  notes  that  NASA  and  the  military  are  exploring  the  use  of  surgical  robots  for  space 
missions  and  combat  zones,  and  that  SRI  is  “providing  help.”  The  article  links  to  a  recent  SJ Mercury > 
News  article  that  highlights  SRI’s  role  in  the  NEEMO  12  mission. 

URL:  http://www.roboticsonline.com/public/articles/articlesdetails. cfm?id=2934 

SpaceRef.com:  NASA  NEEMO  12  Mission  Journal  Tuesday,  May  8,  2007 

This  is  a  journal  entry  regarding  NASA’s  NEEMO  12  mission.  A  demonstration  of  SRI’s  M7  surgical 
robot  is  among  the  upcoming  experiments  described  in  the  journal. 

URL :  http  ://www.  spaceref.  com/news/viewsr.html?pid=24 1 5 7 

Tennessean:  Telemedicine  Could  Allow  Remote  Surgery  by  Robot  Someday 

This  article  is  about  telemedicine  breakthroughs  on  display  at  the  American  Telemedicine  Association 
conference  in  Nashville.  The  article  describes  an  SRI  demonstration  where  an  engineer  in  the  exhibit  hall 
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controlled  a  remote-controlled  surgical  robot  underwater  off  the  Florida  Keys.  The  article  includes  a  photo  of 
Tom  Low,  director  of  SRI’s  medical  systems  program,  performing  the  demonstration. 

URL:  http://www.tennessean.com/apps/pbcs.dll/article7AniW20070515/BUSINESS0 1/705 150333/1 003 
Also  See: 

Ashland  City  Times 

URLhttp:  //www  .ashlandcitytimes.com/apps/pbcs.dll/article? AID=/2007 05 1 5/BUS1NESS0 1/705 1503 3 3/1 436/ 
BUSINESS 

Robertson  County  Times 

http://www.rctimes.com/apps/pbcs.dll/article?  A I  D=/200705 1 5/BUSI  N ESSO  I /705 1 50333/ 1 436/BUSI  NESS 

Fairview  Observer 

http://www.fairviewobserver.com/apps/pbcs.dll/article?  A!  D=/20Q7 051 5/ BUSINESSO  l /7Q5 1 50333/ 1 436/ BUS 

1NESS 

Space.com :  Undersea  NASA  Expedition  a  Success 

This  article  reports  on  the  successful  completion  of  NASA’s  NEEMO  12  undersea  mission  and  notes 
that  SRI’s  “M7  surgical  automaton”  was  studied  for  future  applications  in  remote  areas  of  the  world 
and  on  long-duration  spaceflights. 

URL:  http://www.space.com/missionlaunches/070518  neemo!2  ends.html 
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Appendix  D 
UW  BRL  Report 

The  appended  report  is  as  it  was  prepared  by  UW  BRL.  It  has  been  fonnatted  to  fit  this  final 

report. 
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Overview 

The  University  of  Washington’s  BioRobotics  Laboratory,  under  the  direction  of  Dr  Blake 
Hannaford,  supported  the  University  of  Cincinnati’s  telesurgery  effort  during  the  NEEMO  12 
mission  conducted  May  7-18,  2007.  The  UW  RAVEN  telerobotic  system  was  deployed  to 
the  Aquarius  Habitat  to  conduct  a  variety  of  research  tasks. 

The  goals  of  this  mission  were  to  advance  and  demonstrate  technologies  related  to  remote 
healthcare  for  astronauts  on  extended  space  missions.  In  particular,  the  capability  of  surgical 
intervention  by  remotely  operated  surgical  robotics.  Two  surgical  robots  were  deployed  into 
the  Aquarius  habitat,  the  UW  RAVEN  and  the  SRI,  International  M7  robot.  Control  of  the 
robots  was  provided  over  an  Internet  link  from  UW  (Seattle,  WA)  and  SRI  (Palo  Alto,  CA), 
respectively.  The  last  10  mile  connection  to  the  remote  site  was  provided  by  a  microwave 
link  from  shore  to  buoy  and  a  20m  cable  down  to  the  habitat.  Organization  for  the  remote 
surgery  component  of  the  NEEEMO-12  mission  is  included  expertise  from  Dr.  Hannaford, 
Dr.  Jacob  Rosen,  Mr.  Mitch  Lum,  and  Ms.  Diane  Friedman. 

Mission  Preparation 

Prior  to  the  mission,  UC  surgeon  Dr.  Timothy  Broderick,  NASA  flight  surgeon  Dr, 
Joseph  Schmid,  and  geologist  Mary  Sue  Bell  came  to  the  University  of  Washington  for  two 
days  of  training.  They  learned  and  practiced  procedures  for  operation,  assembly  and 
disassembly  of  RAVEN. 

In  preparation  for  integration  of  the  RAVEN  into  the  habitat,  NASA  requirements 
necessitated  creation  of  extensive  operational  documentation  for  the  RAVEN’s  startup, 
shutdown,  and  E-stop  recovery  procedures.  RAVEN  was  shipped  to  the  NURC  operational 
facility  on  23 -April-2 007.  Two  UW  graduate  students,  Mitch  Lum  and  Diana  Friedman  then 
traveled  to  NURC  and  set  up  RAVEN  and  verified  operation.  The  RAVEN  was  then 
dismantled,  packaged  and  potted  down  to  the  Aquarius  by  U.S.  Navy  divers.  Mr.  Lum  made 
a  dive  down  to  the  habitat  to  repair  some  damage  to  the  RAVEN  sustained  by  water  pressure 
applied  through  the  walls  of  the  dive  bag. 

Drs.  Broderick  and  Schmid  reassembled  the  RAVEN  in  Aquarius  for  the  teleoperation 
experiments.  (Figure  1)  The  RAVEN  was  controlled  from  three  separate  locations.  Master 
consoles  were  set  up  in  Seattle,  at  the  shore  base  in  Key  Largo  FL,  and  at  Cincinnati  Museum 
Center  in  Cincinnati,  OH.  Operators  at  these  locations  tested  other,  non-surgical 
performances  of  the  robot  including  simulated  manipulation  of  moon  rocks  in  a  sterile 
environment.  All  locations  used  the  Sensable  Phantom  Omnis  as  the  input  devices.  (Figure 
2).  Surgeons  performed  experimental  benchmark  tests  drawn  from  the  Society  of  American 
Gastrointestinal  Endoscopic  Surgeons  (SAGES)  Fundamentals  of  Laparoscopic  Surgery 
(FLS)  test  protocol  (Figure  3). 

For  surgery  tasks  in  Seattle,  two  different  video  systems  were  used  (HaiVision  to  VLC 
and  iChat  V.2.1.3  on  Apple  Macintosh),  as  explained  below.  The  HaiVision  1060  (HaiVision 
Systems  Inc,  Montreal)  is  a  hardware  video  CODEC  providing  MPEG-  4  AVC  H.264  video 
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compression  and  decompression.  The  HaiVision  1060  encoded  the  video  stream,  and  a  laptop 
PC  running  VLC  media  player  (http://www.videolan.org/vlc/)  displayed  the  video  on  the 
surgeon  side.  Picture  quality  was  excellent  at  full  laptop  screen  resolution.  However,  latency 
between  Seattle  and  Florida  was  quite  noticeable  to  users,  on  the  order  of  one  second. 
Internet  round  trip  latency  for  the  command  packets  was  measured  as  only  70  ms  so  the 
majority  of  this  time  was  due  to  video  compression  and  decompression.  The  HaiVision  video 
parameters  for  the  NEEMO  12  experiment  are  shown  in  Table  2.  Video  picture  quality  using 
the  HaiVision  system  is  approximated  by  the  screen  photo  taken  during  the  mission,  Figure 
4. 


Figure  1.  RAVEN  robot  set  up  inside  the  underwater  habitat  and  positioned  for 
experimentation  with  the  laparoscopic  training  tasks. 
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Figure  2.  Portable  surgical  workstation  as  set  up  at  the  NEEMO  on-shore  site  showing 
Phantom  Omni  hand  control  devices  and  video  feed  from  the  underwater  habitat. 


Figure  3.  Control  station  set  up  at  UW  during  NEEMO  mission.  Dr.  Andrew  Wright  is 
operating  the  robot. 
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Table  2.  Video  Parameters  for  NEEMO- 12  Experiment 


Parameter 

Value 

Encapsulation 

H.264 

Video  Input 

S- Video 

GOP 

30 

Framing  Mode 

IP 

Interlacing 

MBAFF 

Resolution 

Full  D1  (720x480) 

Video  Bit  Rate 

1-1.5  Mbps 

Figure  4.  Screen  shot  taken  during  the  NEEMO  experiment  showing  video  quality 
obtainable.  This  photo  was  taken  from  the  ViewSonic  LCD  monitor.  Color  balance  has  not 
been  measured  or  corrected. 

RAVEN  Surgical  Robot  Mechanism  and  Control 

The  7-DOF  surgical  manipulator  used  force/torque  data  collected  by  the  Blue  Dragon,  the 
preliminary  experimental  evaluation  and  the  kinematic  optimization  as  a  foundation  for  its 
design.  The  robot  is  divided  into  three  main  subsystems:  the  static  base  that  holds  all  seven 
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actuators,  the  spherical  mechanism  that  positions  the  tool,  and  the  tool  interface.  The  motion 
axes  of  the  surgical  robot  are: 


1 .  Shoulder  Joint  (rotational) 

2.  Elbow  Joint  (rotational) 

3.  Tool  Insertion  /  Retraction  (linear) 

4.  Tool  Roll  (rotational) 

5.  Tool  Grasping  (rotational) 

6.  Tool  Wristl  Actuation  (rotational) 

7.  Tool  Wrist2  Actuation  (rotational) 


Figure  5.  RAVEN  CAD  model  and  RAVEN  during  animal  experiment. 

The  first  four  joint  axes  intersect  at  the  surgical  port  location,  creating  a  spherical 
mechanism  that  allows  for  tool  manipulation  similar  to  manual  laparoscopy.  Brushless 
motors  mounted  to  the  base  of  the  micromanipulator  actuate  all  motion  axes.  The  motors  are 
mounted  on  quick-release  plates,  which  allows  for  motor  removal  without  the  need  for 
disassembling  the  cable  system.  Maxon  EC-40  motors  with  12:1  planetary  gearboxes  are 
used  for  the  first  three  axes,  subject  to  the  highest  torques.  Maxon  EC-32  motors  are  used  for 
the  remaining  axes.  Maxon  DES70/10  series  amplifiers  drive  the  motors. 

The  selection  of  DC  brushless  motors  over  brushed  motors  was  motivated  by  a  better 
torque  to  weight  ratio  as  well  as  more  efficient  heat  dissipation  due  to  the  fact  that  the 
motor's  windings  are  thermally  coupled  to  its  outer  case.  While  the  performance  benefits  of 
brushless  motors  are  clear,  they  required  more  complex  and  expensive  controllers  and 
extensive  wiring  (14  conductors  per  motor).  The  motors  of  the  first  three  axes  have  power- 
off  brakes  to  prevent  tool  motion  in  the  event  of  a  power  failure. 
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The  cable  system  is  comprised  of  a  capstan  on  each  motor,  a  pretension  adjustment 
pulley,  a  pulley  array  to  redirect  the  cables  through  the  links,  and  attachment  to  each  motion 
axis.  The  shoulder  axis  is  terminated  on  a  single  partial  pulley.  The  elbow  axis  has  a  dual¬ 
capstan  reduction  stage  terminating  on  a  partial  pulley;  the  tool  insertion  /  retraction  axis  has 
direct  terminations  of  the  cables  on  the  tool  holder.  The  tool  rotation,  grasping  and  wrist 
cables  are  terminated  on  capstans  on  the  tool  interface  box. 

Each  axis  is  controlled  by  two  cables,  one  for  motion  in  each  direction,  and  these  two 
cables  are  pre-tensioned  against  each  other.  The  cables  are  each  terminated  at  both  ends,  to 
prevent  any  possibility  of  slipping.  The  cable  system  maintains  constant  pretension  on  the 
cables  through  the  entire  range  of  motion;  however,  there  are  force  and  motion  couplings 
between  the  axes,  which  must  be  accommodated  for  in  the  control  system. 

The  mechanism's  links  are  machined  from  aluminum,  and  are  generally  I-section  shapes 
with  structural  covers.  These  removable  covers  allow  for  access  to  the  cable  system,  while 
improving  torsional  stiffness  of  the  links  when  they  are  in  place.  The  links  are  also  offset 
from  the  joint  axis  planes,  allowing  for  a  tighter  minimum  closing  angle  of  the  elbow  joint. 
Laser  pointers  attached  the  shoulder  and  elbow  joints  allow  for  visual  alignment  of  the 
manipulator  relative  to  the  surgical  port.  When  the  two  dots  projected  on  the  skin  of  the 
patient  converge,  the  manipulator  is  positioned  such  that  the  center  of  rotation  of  the  surgical 
manipulator  is  aligned  with  the  pivot  point  on  the  abdominal  wall. 

Each  surgical  manipulator  has  a  mass  of  approximately  15  kg,  which  includes  the  motors, 
gear  heads  and  brakes. 

Tool  Interface: 

The  tool  interface  (Figure  6)  allows  for  quick  changing  of  tools  and  controls  the  tool 
rotation,  grasp  and  wrist  axes.  A  robotic  tool  changer  can  release  the  tool  from  and  attach  the 
tool  to  the  mechanism  with  a  single  grasping  motion.  The  tool  tips  used  are 
modified  micro-wrist  tools  from  the  Computer  Motion’s  Zeus  platform. 

The  tools'  grasp  and  wrist  axes  are  actuated  by  pushrods  in  the  tool  shaft. 

High  pitch  threads  are  used  to  convert  the  rotational  motion  of  the  cable 
system  capstans  into  linear  motion  of  the  tool  pushrods. 

Figure  6.  Tool  interface  allows  rapid  change  of  surgical  tools 

We  developed  a  USB  2.0  interface  board  that  serves  as  the  data  link 
between  the  control  software  (running  on  a  RTAI  Linux  computer)  and  the 
motor  controllers.  The  USB  board  includes  eight  channels  of  16bit  digital 
to  analog  converters  for  control  signal  output  to  each  motor  controller  and 
24bit  quadrature  encoder  readers.  The  board  can  perform  a  read/write  cycle 

The  overall  architecture  is  very  modular  to  facilitate  future  development  and  expansion  or 
collaboration  with  other  groups.  The  master  (Surgeon  Console)  and  slave  (Surgical 
Manipulators)  communicate  through  a  Networking  Module.  Thus,  the  Surgeon  Console 
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could  be  any  piece  of  software  with  supporting  hardware  to  communicate  with  the  slave. 
Figure  7  illustrates  the  overall  software  architecture  of  the  Surgical  Robot  System. 

Figure  7.  Control  System  Architecture 

When  surgeons  transitioned  from  open  surgery  to  MIS,  their  sense  of  touch  was  greatly 
diminished.  Friction  between  the  surgical  tool  and  the  seal  of  the  MIS  port,  as  well  as  the 
forces  exerted  on  the  port  by  the  abdominal  wall  greatly  reduce  the  surgeons'  ability  to  feel 
the  tissue  they  interact  with.  With  the  right  master  interface  device,  surgical  robots  present  an 
opportunity  for  surgeons  to  regain  and  potentially  enhance  their  sense  of  touch  in  MIS 
procedures.  One  of  the  design  goals  was  to  develop  a  lightweight,  low  inertia  and  back- 
drivable  mechanism  suitable  for  enabling  force-feedback  capabilities  in  bilateral 
teleoperation. 

Currently,  the  Phantom  Omni  (Sense Able  Technologies)  is  used  as  the  master  device. 
The  Phantom  line  of  SenseAble  haptic  devices  is  well  established  among  haptics  researchers. 
The  cost-effective  Omni  provides  a  straightforward  software  interface  that  allowed  rapid 
implementation  of  a  surgeon  interface  device  into  a  master/slave  system.  It  features  3-DOF 
force-feedback  and  6-DOF  position/orientation  sensors.  Additionally,  the  surgeon  station 
includes  a  position  indexing  foot-pedal  that  enables  and  disables  the  surgical  manipulator, 
allowing  the  surgeon  to  put  down  the  Omni  styluses  without  moving  the  robots. 

Software  and  Safety  Architecture: 

The  control  system  and  supported  electronic  hardware  were  designed  to  incorporate 
safety,  intelligence,  modular  design,  and  flexibility.  As  this  is  a  medical  device,  the  most 
critical  of  these  aspects  is  safety.  Inherent  to  a  safe  system  is  robustness,  reliability,  and 
some  level  of  automatic  override.  Our  system  includes  safety  features  such  as:  a  small 
number  of  states,  Programmable  Logic  Controller  (PLC)  state  transition  control  (Figure  8), 
active  enable,  power-off  brakes,  E-STOP,  and  a  surgeon  foot  pedal.  With  these  features  we 
have  a  system  that  we  expect  will  provide  a  level  of  predictability,  reliability,  and  robustness 
sufficient  for  animal  surgery  evaluation. 


Figure  8.  Control  System  State  Diagram. 


A  programmable  logic  controller  (PLC  -  Direct  Logic  05)  controls  the  system  state 
transitions  based  on  inputs  received  from  the  system  and  generates  outputs  to  control  the 
RTAI  software  state,  the  motor-enable  and  brakes.  PLCs  are  a  well-developed  technology 
used  extensively  in  automation  applications.  PLC  technology  is  assumed  to  be  inherently 
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reliable,  providing  built-in  safety  circuitry  that  is  easy  to  use.  In  addition  to  monitoring  the 
system  hardware,  the  PLC  monitors  the  health  of  the  control  software  through  the  use  of  a 
watchdog  timer.  The  watchdog  timer  monitors  a  square-wave  signal  from  the  control 
software.  This  square-wave  may  be  considered  the  “heartbeat”  of  the  control  software.  In  the 
event  of  a  software  failure,  the  PLC  will  detect  the  loss  of  the  heartbeat  signal  and 
immediately  switch  the  system  into  the  Emergency  Stop  state,  enabling  the  brakes  and 
disabling  the  motors.  An  array  of  status  LEDs  display  the  current  state  of  the  system. 

Engineers'  Interface 

Robot  development  is  assisted  by  the  Engineering  Interface  (El);  a  low-level  interface  to 
the  system  states  and  control  software.  Developers  are  presented  with  an  intuitive  GUI  with 
easy  access  to  the  robot's  features.  In  development  stages,  the  system  state  -  stop,  init,  run,  e- 
stop  -  can  be  set  manually  with  the  click  of  a  button.  Control  commands  can  be  sent  to  any 
degree  of  freedom  or  the  entire  robot;  for  example,  a  40  degree  sine  wave  can  be  commanded 
to  the  shoulder  joint,  motor  controller  number  two  can  output  30%  maximum  current,  or 
endpoint  position  can  be  instructed  to  move  3cm  to  the  left.  Furthermore,  robot  states  (i.e. 
motor  output,  joint  position,  end-effector  position)  are  output  on-screen  in  real-time,  and  also 
logged  for  later  evaluation. 

The  El  can  connect  to  the  RTAI  Linux  control  system  by  two  FIFO  device  nodes  or  a 
single,  bi-directional  (TCP/IP)  network  socket.  Two  types  of  data  are  exchanged:  (1)  a  packet 
containing  all  robot-state  information  is  received  by  the  El,  and  (2)  a  command  packet  with 
all  instruction  parameters  is  sent  from  the  El  to  the  control  software.  This  link  is  independent 
of  the  master-slave  link.  A  touch-screen  based  surgeon  interface  has  also  been  developed. 

Research  Findings 

Publications 

The  following  publications  describing  some  of  our  NEEMO  results  have  been  published  or 
will  soon  appear: 


•  Mitchell  J.H.  Lum,  Diana  C.W.  Friedman,  Hawkeye  King,  Ganesh 
Sankaranarayanan,  Jacob  Rosen,  Timothy  J.  Broderick,  Mika  N.  Sinanan,  Blake 
Hannaford,  "Raven  -  A  Surgical  Robot  for  Teleoperation",  Submitted,  American 
Telemedicine  Association  Conference,  Seattle,  April  2006. 

•  M.  Lum,  et  ah,  “Objective  Assessment  of  TeleSurgical  Robot  Systems:  Telerobotic 
FLS,”  In-Press,  proceedings  of  MMVR  2008,  Long  Beach,  CA,  January  2008. 

•  G.  Sankaranarayanan,  L.  Potter,  B.  Hannaford,  'Measurement  and  Simulation  of 
Time  Varying  Packet  Delay  with  Applications  to  Networked  Haptic  Virtual 
Environments,'  Proceedings  of  Robocom  2007,  Athens,  Greece,  October  2007. 

•  G.  Sankaranarayanan,  H.  King,  S.Y.  Ko,  M.J.H.  Lum,  D.  Friedman,  J.  Rosen,  B. 
Hannaford,  'Portable  Surgery  Master  Station  for  Mobile  Robotic  Telesurgery', 
Proceedings  of  Robocom  2007,  Athens,  Greece,  October  2007. 
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Lendvay  by  remote  control  between  the  University  of  Washington,  Seattle,  and  the  RAVEN 
robot  situated  in  Aquarius. 

Robot  Motion  Data  Recordings 

Our  computers  collected  about  6.8Gb  of  motion  data  including  position  velocity,  and 
torque  for  both  RAVEN  manipulators  recorded  at  100Hz  for  all  operations  in  Aquarius  and 
NURC.  This  data  is  still  being  analyzed.  A  sample  of  the  data  is  reproduced  in  Figure  9.  In 
this  sample,  the  surgeon  is  moving  the  blocks  up  and  down  on  the  pegs  of  the  FLS  task 
board. 

Network  Data  Performance  Data 

One  of  our  goals  in  the  NEEMO- 12  mission  was  characterization  of  the  network  quality 
between  the  control  site  at  UW  in  Seattle  and  the  Aquarius  habitat.  This  delay  was  composed 
of  two  components:  1)  The  regular  Internet  between  UW  and  NURC  and  2)  the  special 
microwave  link  between  NURC  and  Aquarius.  These  links  were  characterized  in  terms  of 
round  trip  time  delay.  Our  computers  inside  RAVEN  contained  a  packet  reflector  program. 
Another  program  at  UW  sent  regular  streams  of  test  packets  to  the  packet  reflector  and 
measured  round-trip  delay  time  for  each  packet.  Although  similar  to  the  standard  ping 
command  measurement,  this  system  used  UDP  packets  of  the  same  type  used  in 
teleoperation. 

Results 

Measured  delay  distributions  are  shown  in  Figure  10.  The  mean  and  standard  deviation 
were  76.5ms  and  5.4ms  respectively.  When  packet  sending  rates  were  decreased  from 
1000Hz  to  10  Hz  these  statistics  were  essentially  unchanged. 
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Figure  10.  Distribution  of  round  trip  time  delays  for  packets  between  UW  and  the  Aquarius 
habitat  taken  during  NEEMO- 12.  Delay  distribution  was  quite  similar  for  100  Hz  and  10  Hz 
packet  repetition  rates. 

Another  parameter  of  Internet  packet  timing  is  the  correlation  coefficient  of  the  time 
delays  of  adjacent  packets.  This  is  a  measure  of  network  congestion  which  ranges  from  0-1.0. 
If  the  correlation  is  close  to  one,  routers  are  queueing  packets  as  they  must  wait  for  links. 
Packet  delay  correlation  was  essentially  zero  (indicating  low  congestion  for  10  -  100  Hz,  and 
rose  to  0.135  for  500Hz.  The  only  significant  amount  of  congestion  was  observed  at  1000Hz 
were  the  correlation  coefficient  was  0.51.  Although  packet  latency  was  not  affected  as  rate 
was  increased  to  1,000Hz,  the  growing  correlation  coefficient  suggests  that  the  network  was 
near  its  bandwidth  limit. 

Tests  from  UW  to  the  NURC  alone  (i.e.  not  using  the  microwave  link  for  the  last  5  miles) 
gave  a  mean  delay  of  75.2ms  and  standard  deviation  of  6.3ms.  Thus  the  microwave  link  to 
the  Aquarius  was  highly  efficient  and  not  a  network  bottleneck. 

Conclusion 

The  RAVEN  was  successfully  deployed  and  demonstrated  in  the  habitat.  Surgeons  and 
researchers  were  able  to  operate  the  robotic  arms  using  the  controllers  linked  across  several 
thousand  miles.  This  included  the  ATA  exhibit  in  Nashville  and  the  CMC  in  Cincinnati,  OH. 
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